I’m in the gray area of the aro/ace spectrum and I don’t think I could uphold a normative romantic relationship. I wouldn’t want to have sex with anyone I’m in a primary partnership with either, even though I feel sexual desire and have a libido - so what I’d want is an open QP relationship in which I’d have nonromantic sex and sensual intimacy with friends who aren’t my primary. Am I polyamorous?
That sounds like a very specific situation you have there - but it’s your call, and your call only, as to whether to identify as polyamorous. Labels are useful in as much as they help you find like-minded people, and help you find the tools to be healthy and honest in your relationships. So find the ones that work best for you, not the ones other people approve of. Good luck!