My guess would be that the fish are just too small and your hooks are too big for their mouths to get a proper hook set.
They could be like a bait fish such as shad or minnows. On the other hand, they could possibly crappie, bluegill, or even crawfish, little stripers or small bass. It depends where you're fishing at and what species of fish are in that water.
My suggestion would be to change out your hook with a smaller hook. Then you should have a better chance of hooking up.
This has happened to me plenty of times. And I'm pretty certain that most bait anglers have experienced this from time to time as well.
In one instance that I had a few years ago, I was fishing in a cove of a reservoir. My kids and I were using worms, and the fish just kept stealing my bait. Finally I had enough so I put on a smaller hook and threw it out there with a bobber. Next thing I know we start catching a whole bunch of bluegill. Although we didn't catch any big giant fish like we hoped for that day, we still had a blast catching one fish after another.
So basically, instead of just providing the fish with a free meal, downsize your hook. Then you should be able to get into some nice action.
Another thing you can try, whether or not you change out your hook, is to pinch off about half of the worm. This way it won't dangle so far from the hook to where it can get pulled away from the hook. The fish will be forced to have that metal in their mouth in order to get a bite to eat.
Hope this helps you out. Feel free to let me know how it goes for you.