Simply because a lot of people think that anything fun, wacky, colourful or absurd is immature and that only a young person with a lot of spare time would prioritize anime over more 'practical' or sophisticated hobbies.
In the West though, I think it's also because a lot of people don't know anime well. They associate animation with western cartoons that are primarily aimed at children or they only know anime through Pokémon and pictures and haven't heard of the darker series.