You probably mean it was denied? Applications get rejected if you pay incorrect fees, submit incomplete or out of date forms. Rejected and denied are very different things. With rejected no decision was made. Denial is a decision by USCIS.
I don't think appeal would work. In the first sentence you said petition was denied because no divorce decree was produced. That's a valid reason for denying petition. Even if you try to challenge the decision, USCIS will simply request divorce decree again. Since nothing changed and you don't have a decree, it will not be approved. To me, a better strategy is to file a new I-751. It will give you more time to finalize divorce and get the decree while it's pending.
No, you cannot get visas while you're LPR. Even if you give up GC, H-1B involves lottery, the outcome is not guaranteed and you won't likely get visa based on your complex immigration history.