I'm an undergraduate studying math and I'm taking a graduate math class for the first time this semester. I've taken the undergraduate prerequisites, but it's still very challenging, which I was expecting.
I'm starting to get overwhelmed by the content and the pace of the course. The professor is not only very famous in the field but also tends to brush off most things as 'basic' during lectures and doesn't seem to like to answer students' questions in detail. I find it intimidating to approach the professor or go to office hours because I feel like I don't even know the basics. I'm the only undergraduate in the class, but it seems like other graduate students are also finding the content quite intense.
I also don't have anyone to work with, and I find it very difficult to just randomly approach people in the class, not only because I'm introverted but mostly because I feel very embarrassed of my lack of understanding. I'm really interested in the subject and I want to give it my best. It's hard to tackle the problem set even after consulting multiple books, and I feel more discouraged and unsure of myself. I'd appreciate some advice from people with more experience.
Mathematician here: We've all been in a similar situation when transitioning from undergrad to grad classes. You write that you are "embarrassed" due to your lack of understanding; don't worry, others probably are having issues also. But if you do think you aren't getting basics, then it is more, not less important that you talk to your fellow students and the professor.
However, the main thing you should realize is that this is a pretty regular experience. What matters is how you react to it; don't get demoralized but put the effort in. And don't hesitate to ask questions, even in class; your fellow students are likely confused about some of the same things you are.
I was also in that exact situation once. As you can see from other answers, many of us were; it's a common experience.
The main thing that I regret, and I think limited my advancement at that exact point, was the inability/unawareness of reaching out to others and forming/joining a study group. (Turns out I was "competing" in some sense with another group of a dozen or so students who studied together.)
There's an actual a name for this phenomenon of feeling alone and spending overwhelming amounts of energy trying to get over that to the point of burnout, called John Henryism. In the literature, it's mostly associated with high-performing African-Americans who feel isolated, but I think it's applicable to a wide range of situations, esp., people from rural areas seeking to succeed in elite academia/industry. It seems particularly likely to afflict personalities who (a) are introverted and tend to go into math, (b) have had high success at the undergraduate level working alone from text resources, and (c) try to make this radical leap to graduate-level math work (not having the experiences or toolbox to get outside help).
In short: Push through and do what you need to join or create a study group. Also, go to the instructor's office hour and lay out your situation and experience. It's my primary regret that I couldn't think to make that happen in the exact same circumstance.
There are a couple of things that the other answers don't mention:
The official prerequisites may not line up with what you are actually expected to know in the course. Being a graduate course, the professor probably expects the students to have a certain amount of mathematical maturity, which is not something that can be accurately measured in terms of prerequisites.
There are different styles of graduate courses. One is where you are taught "basics," but at a higher level/faster pace than undergraduate courses. (These often have regular homework and/or exams.) Another type is a what I think of as a more "seminar-style" type course, where you are trying to teach something broader or deeper than a basic course, and you don't have time to go through all the underlying details to cover the material. (These ofter have no graded homework or exams.) Then there are things in between.
I can't tell from what you've said which type of class yours is, or if it's being taught in a reasonable way, or if the other students are following along better than you are. Depending on your situation, it might be better to switch to another undergraduate elective if possible.
But if you want to diagnose the situation, and do better in the class, try studying with some of the other students, and try going to office hours and asking the professor about some of the details. Even if they're not willing to explain all the details to you, they should at least be willing to say tell you whether you shouldn't be in this class if you don't know this already or suggest references where you can learn how to fill-in the details for yourself. It's also possible the professor doesn't know you (and the other students maybe?) don't know these things and this will help make things clear. (I often don't know what my students are familiar at the beginning of a course.)
You definitely want to talk to the professor during office hours and be open about your struggles in the class. They will be able to give you some advice and direction about how to tackle the material. Make sure to mention you are an undergraduate so the professor knows you are coming from a different background than the other students.
Don't feel embarrassed by your lack of understanding, I guarantee there are other students who barely understand it themselves or are out of practice from when they took other mathematics courses years ago during their undergraduate study.
Have you asked yourself what you want from this course?
You haven't mentioned how far along you are in undergrad, nor how many weeks into the semester you've progressed. I applaud you for taking the challenge. Do not be demoralized! Some graduate classes and materials are brutal for nearly all students.[*] It certainly was the experience in my case.
That being said, you need to have a conversation with faculty and perhaps, importantly, yourself. Really ask yourself if the time you need to invest to push through the material is worth the payout here. Whatever material is contained in the course will not disappear from the annals of history - you can revisit such a course in the future. Again, I'm not sure how far into the semester you've gone, nor what your needs are, but consider such questions as:
Is the course required? Will it help you achieve your goals? What are your goals? How does the energy necessary for this course weigh against your other responsibilities and interests?
Throughout this analysis, remember: Do not be afraid to turn to help. And, do not be afraid to drop the course. I mention the latter because it is not emphasized in the other answers. I'm not questioning your ability - no! I'm questioning whether you'll be happy after allocating your time to push through the material. Don't fall victim to the sunk cost fallacy, nor to your whims. It's tough to cut projects, yes, but there is no shame in it. It's a skill actually! Keep your options open.
[*] Consider that grad students often have varied levels of preparation. Some may find the course harder than you are. You're not alone in the struggle.
Talk to your professor, fellow students, tutors. The department almost surely has resources to help students learn, and it's up to you to use them. As long as you make a genuine attempt to learn the material and don't show up expecting to be spoonfed, virtually everyone will be willing to help you.
As for being embarrassed: this isn't something to be embarrassed about. In fact if anything it would be embarrassing if the material were so simple you can understand it easily. Besides, would you rather be embarrassed by your lack of understanding now, or by an F grade at the end of the semester? Being part of a university gives you access to resources an outside self-studying student doesn't have - make use of them.
I was in a similar situation myself as an undergrad (physics), as I took an advanced quantum field theory course designed for graduate students. It was also taught by two lecturers - one of whom was excellent at teaching, the other less so...
At the time I had a good relationship with a professor who had previously taught the course, with whom I was able to talk about the subject matter. I also spent time with the graduate students on the course, both of these strategies are immensely useful and can help you figure out what you need to figure out.
In addition looking at other resources (particularly useful if the subject matter is a commonly taught subject) such as lecture notes and problem-sheet solutions was very helpful when I came across a subject I didn't feel I understood.
It is also very useful (in my experience) to look through problem sheets/past exams and make sure you understand the problems on there, as these are often the intended key take-aways and important points (it can be tough to sort the unnecessary detail in very complex subjects!).
Wishing you the best of luck!
I agree with the answers here, and would add the following points from personal experience (both as a student and as a mentor):
During studies (CS with maths as side dish), I had this effect twice.
So: a) check if what you are listening to is really "for you", i.e. whether it actually is something you are kind of familiar with, or really want to become familiar with, or if it is some weird stuff that does not really connect with you. And b) check how much effort the more successful students are investing. Maybe it is one of those courses that views the classroom work as more of a hint of what to study on your own, maybe you are supposed to invest the majority of your efforts in private.
Obviously, you'll have jump your own shadow to talk with your co-students; I guess it's up to you whether you are able to deepen the work you do together with them. But that's maybe more a topic for the interpersonal StackExchange...