Need to Port to Python3

[kmitch95120]

I just spent an unreasonable amount of time getting staticDHPCd running on an Ubuntu 20.04 server and I'd have much rather spent that time helping to port this project to Python3.

What will it take to get this going?

[flan]

I don't anticipate that the porting process itself would be terribly hard (and some parts of libpydhcpserver could be simplified by use of the ipaddress module), but the barrier, beyond time, is testing. I don't have a particularly large network with which to do validation or many systems to which I could potentially introduce chaos, so I'd need to delegate validation to others and the rate of anyone passing back even a "yeah, this works" has been pretty close to zero.

The largest technical hurdle is probably custom modules, since the import semantics changed between Python 2 and 3 and it'll probably take some experimentation to get the search-paths lined up properly so it feels intuitive to use, plus the lack of backwards-compatibility that it'll bring.

[kmitch95120]

I'd suggest we create a 2.0.0 branch for Python2 and also a 3.x.x. branch that requires Python3. It's convenient that the staticDHCPd version numbers align with the major Python versions. Is that a sign? As for testing, I'm signing up. As for others not reporting back, people have a tendency not to say anything when things work. They only complain when they don't.

As for backwards compatibility, I'd suggest we keep the 2.0.0 branch for those who wish to use Python2 (at least until they can't get it installed anymore) and also offer some 3.x.x conversion guidance for those with custom modules.

Are you still using your redhamsterx email address? I have that from many years ago when we worked on a Dell PowerEdge IDRAC dhcp issue.

[flan]

That's pretty much what I was planning to do when the time eventually came to do the Python 3 port.

I'll try to get started on it this weekend, but I'm not sure how much time I'll be able to dedicate to it on a regular basis.

Given my recollection of the code, it shouldn't take too long to have something that's testing-ready prepared, assuming a 1:1 conversion without using any new Python 3 features, but it will take about as long to enumerate everything that needs to be tested so it can be checked off.

[flan]

And yeah, any e-mail addresses you may have for me are still usable. Unless you need to keep some information confidential, though, I'm fine with keeping the discussion public, in this thread or others.

[flan]

Work on the 3.0.x branch has started.
I think I got everything except potential byte/str conflicts addressed in libpydhcpserver today and I'll chip away at staticDHCPd over the week. I'm thinking I'll be able to do some preliminary testing next weekend, but I don't feel comfortable committing to that target at this time.

[kmitch95120]

Great News! I've started defining test cases and I plan to implement them using scapy. I've also asked one of our support engineers for help in creating a stress test environment using our new workload generator engine called StackPlay. We'll start with 2500 clients and go from there.

[flan]

I didn't get as far as testing, since it was a 96-hour work-week, but I might be able to run it in some capacity tomorrow.
TODO: revise extensions, dhcp.py, and config.py, the latter of which will be the most difficult part because of the import semantics; the others are almost all just logging syntax changes and exception-handling.

I think the testing list near the top of README.md covers everything that needs to be verified before reaching a release-candidate state.

[flan]

It's now in an "it runs" state and the web interface displays logs, so I think, syntactically, the code is Python3-compliant.

But I'm hitting str/bytes mismatches in all the places I expected to encounter them, so I'd recommend against trying it right now unless you want to send a pull-request with a bunch of .decode('utf-8')s. I'll probably have time to get it processing packets on Wednesday, if not tomorrow, but it's all subject to work-related demands on my time.

After that, it'll be getting extensions working and documenting any changes in configuration flow for migration, then testing can begin in earnest.

[flan]

It's now in a state where the makedebs script runs and produces packages that can serve, renew, and release leases, so core functionality is verified.

I haven't yet touched extensions, so if you have any custom logic that needs to be imported (rather than defined in-line), I'd expect it to fail. Otherwise, config should be identical to 2.0.x and still support 1.0.x flavours as well, provided any site-defined functions in there are updated to Python 3.

If you don't depend on extensions, you could probably start experimenting with it and seeing if you can cross off any of the testing targets. I'd expect there to be a lot of small errors, though. If you can correct any of them, I'll take pull-requests. Otherwise, pointing me in their direction will help because my testing vectors aren't going to focus on databases until all of the runtime datastructures and hooks are validated, but I can probably fix errors in them quickly if they're flagged.

[flan]

Extensions should now work, though you'll have to make minor changes to conf.py. See the "3.0 errata" section on the 3.0.x branch's main page for information.

I'll try to cross off a testing item or two every day, provided I can set up a suitable environment for them.

[kmitch95120]

Good to hear. I'm buried at work this week but should be able to start testing this weekend. I've setup a Proxmox cluster for isolated testing.

[KevinRiordan]

hey all, I am considering using this tool to conduct what is essentially dhcp relay testing. the callback functions and other features seem to meet my needs. I am a little confused regarding the versions though, and need the tool to be python3 compliant. the releases in the README only show versions 1.x and the documentation refers to 2.x and now you guys are talking about 3.x.

cloned the repo and going to try and play around with it. I would be happy to provide feedback as needed. hopefully everything goes smoothly

[kmitch95120]

Hi Kevin,

The 3.0.x branch supports Python 3. After you clone the repo you'll need to switch to the 3.0.x branch.

Let us know if you need a hand.

[flan]

2.x never saw a formal release because I felt uncertain about calling it "ready" with some untested features. They were implemented in response to tickets, but I never received confirmation that they worked -- the other party just went silent at the end, so it ended up dangling.

I'm planning to be more aggressive with calling 3.0 ready when I'm mostly confident stuff won't catch fire, since it's become apparent that people are more likely to use it (and therefore report issues) if they see something in the releases list or don't see "release candidate" for years on end, even when it's probably perfectly stable and production-ready.

I would still greatly appreciate confirmation that anything in the to-be-tested section at the top of the README works for you, though. And any 2.0 documentation and discussion should still apply to 3.0. Aside from the caching-related issues that are still open, which I'm working on this weekend, I'm intending for it to be as close to identical as possible, just targeting Python 3.5+ instead of 2.3+.