neighbors

I can't decide if we are just the worse neighbors or we keep getting unpleasant people. At the other house we had the wicked witch (WW) who complained cause the boys would throw things over the fence occasionally. Now the time they were throwing rocks yeah they were being bad, but a few toys seems like little to be upset over to me.
Now the new neighbors who have lived here less than 2 weeks have been over to complain about noise. Now a little background on our house we live in a two-story duplex downstairs there is a bathroom that touches their bathroom and the staircases touch as well as one of the kid bedrooms and the master bedroom. On Monday the kids were a bit out of control as it was Presidents Day they were home from school and were playing in the downstairs bathroom around 10:30. The guy came over and said something about how his daughters room is just over the bathroom (our house is mirrored to theirs so we know the layout) and that it was really noisy. Yes they were being noisy I was sitting here watching them just play but it only lasted a few minutes before I told them to knock it off and move on, which was before he came to complain. So they were noisy for 5-10 minutes and then quiet and he came to complain. Tony answered the door and the guy introduces himself with his rank and last name. It's not like we can't read the door just like you can. Surprise Surprise we're the same rank just like the door says. Anyway. He then said some story about where the kids room is located and Tony said we'd be more quiet.
Today I had gone to the grocery store and had asked Tony and the bigger boys to come help carry groceries in and we all made a few trips in and out and then shut the door. I went to attend to Abbie in the kitchen (big open floor plan) and Tony went to put some groceries away in the laundry room (right off the kitchen). After about 2 minutes I asked Zavier why the front door was open and he and Zachary told us that the neighbor had knocked on it. Well by this time he was gone, so we don't really know what he wanted. I'm only guessing it was to complain about the noise. I guess the noise from carrying in groceries. I don't know.
I like our house right now and where we are. I want to be happy in life, but this kind of friction makes me crazy. While my husband can just move on with his simple brain, my complex brain can't. I don't want to have to tip toe around for the rest of our lives. I feel like our little boys will never just get to be little boys in their own space.
We were using swimming as a family activity, but after Monday when the lifeguard wouldn't let us swim in the big pool with all 5 kids it is just less than appealing in the kiddie pool where the water is constantly being dumped on your head by the toys in the area. And I am more afraid Zander will drown in the kiddie pool than the big one with or without a life jacket.
I just don't know what to do with my kids. It seems like everywhere I turn we're doing something wrong as a larger family.