@ibowankenobi has it right, but let's use the right terminology. When storing SIGNED numbers in Binary (which is all JS lets you use) the top-most bit is the sign.. When a number is signed it is stored as the "two's compliment" of the number, which is to say the 'data' bits are reversed.
This is done so that all addition and subtraction works right. Let's say we were dealing with 8 bit signed just to keep the lengths under control. When you go past zero or the integer limit, the math has to 'roll over' (also setting the carry bit).
To that end:
0b00000000 - 1 == 0b11111111 == -1
Since we want zero minus 1 to be negative 1. Just as 0b1000000 is negative 128, not positive. (again for 8 bit)
Which is why you can't just flip the negative bit of your data to make the number negative, NOR can you simply use not.
You want it unsigned, mask off the sign bit... with JavaScript it's 32 bit, so state that as hex.
var num2 = ~num1 & 0x7FFFFFFF;
Mask off the sign from the result you'll get what you expect for unsigned not.