WCAG compliance and/should actually have VERY little to do with layout. The reason is that most of the users for whom the things the norms would violate shouldn't get the mobile menu or even your graphical layout in the first place.
So long as the underlying markup is semantically correct, it doesn't matter to users of screen readers, braille readers, or search engines what tricks you pull in terms of positioning or scripting.
BUT
There are things that can screw that up. These things apply in equal measure to not just mobile/hamburger menus, but also conventional dropdown menus.
The biggest is display:none. Search engines, some 'readers', and other various UA's will ignore content set to display:none; because the black-hat SEO scammers abused it fifteen years ago for "content cloaking" -- padding their pages with search words hidden from the user to try and game the search engines.
It is for this reason that using absolute positioning to slide the hidden element off the screen, or chopping it off with overflow:hidden came into being used for dropdown menus.
Now that said, if you're sending your stylesheet setting display:none as media="screen" in the stylesheet LINK, it will NOT be an accessibility violation and should be ignored by everything except search. So it's not an accessibility woe, it's an SEO one that could cause issues spidering your site. One of the many reasons people get duped into thinking sitemaps are legit and serve a purpose.
The second issue is one that plagues a lot of code out there, a lack of graceful degradation thanks to implementing them using JavaScript to do what HTML and CSS can do without it.
If we're talking mobile menus... well, take Bootcrap's implementation. They put a button tag and a bunch of "span for nothing" into the markup and the entire thing REQUIRES JavaScript to function.
JavaScript in and of itself is not an accessibility violation, but navigation (or content loading) that only works via JS with no graceful degradation plan most certainly is.
But again, that's a nasty case of "JS for nothing and your scripts for free. That ain't workin', that's not how you do it.".
See, if we're talking about mobile menus that kick in thanks to responsive layout, that means media queries. Every browser that supports media queries supports the :target and :checked CSS pseudo-states, meaning you don't need JavaScript for this!!!
The method I favor uses a nameless valueless checkbox with an associated empty label. I slide the INPUT off screen, create the hamburger with generated content, then in the CSS just use the adjacent sibling selector to target everything.
It's far more accessible and less of a hassle, with most (but not all) alternative UA's actually ignoring the checkbox since it has no name or value, and the LABEL tag being empty.
There's only one major drawback -- non-CSS users get a checkbox that doesn't do anything. You can hide it for any CSS target you want, but not for non-CSS. Whilst I would consider that a problem, it's no different from the empty BUTTON tag that DOES render CSS off in the scripted approach such as that used by Bootstrap.
One confusing element that doesn't do anything for a minuscule audience that doesn't otherwise negatively impact content delivery? I do not consider that a problem, nor would any measure of accessibility.
See my tutorial on doing so using :checked and an INPUT, comparing to what BS does for markup along the way.
cutcodedown.com/tutorial/mobileMenu
... and it certainly isn't in violation of any accessibility guidelines. The same techniques can be used for modal dialogs, and with CSS animations you can even make them fancy and pretty.
See this login modal that uses :target with hash links instead of JavaScript.
cutcodedown.com/tutorial/modalDialogs
Also not a violation since if the required CSS isn't supported, the form appears at the bottom of the page and clicking on the 'show modal' will simply trigger the normal hashlink behavior scrolling to it.
Good way to avoid accessibility headaches? Write everything to work without JavaScript first, then go back through and enhance it with JavaScript to improve the experience.
That's just part of progressive enhancement. You start at the lowest common denominator (plaintext of your content) and you slowly add technology on top of technology atop it. Doing so in a manner where if any one of the steps along the way goes missing -- images, scripting, css, even html behaviors -- the result is still useful to the end user.
As such if you have the proper semantic markup of the menu, if it can still be accessed and used when scripting is blocked/disabled/unavailable, if it can still be used when fancy stuff like media queries or CSS3 effects go missing... then you're golden.
Of course when people start out from PSD's, or with frameworks that completely bypass or ignore good practices, of course the result ends up in violation.
But few people want to hear that, fewer still are ready to accept it.