Cleaning up, and getting organized
jac_goudsmit
Posts: 418
Hi guys,
I've read most of the threads in this forum and I'm happy to see there are so many good ideas about possible improvements to the P1V. I would like to see all those ideas in a single place, so here's what I've been up to.
I created my own Github fork (https://github.com/jacgoudsmit/P8X32A_Emulation) based on Mindrobots' fork because he added the BeMicro CV tree to Heater's repo, and the BeMicro is the board that I ordered to play with this (remaining stock at Arrow is now down to 3 so get them while you can, hehe). I noticed there was a large number of intermediate files and output files in the repo, which shouldn't be there (they can always be re-generated by doing a build) so I cleaned things up and entered a pull request to MindRobots.
I also noticed there's a lot of redundancy: all Verilog files are repeated in each of the three target directories, and they are mostly identical. That's a problem because if someone would want to implement some cool feature in one of the Verilog files, they would have to remember to change the files in each and every directory. This doesn't agree with the DRY principle (Don't Repeat Yourself) and since my board isn't here yet, I figured I might as well spend my time on taking care of fixing that. I'll probably just make three (for now) top files with different names, all using a single copy of the Verilog files.
I'm also thinking of creating a simple system to conditionally compile features such as port B, larger hub memory, character ROM etc. That way someone in the future can simply download one repository and build it into the standard P1V for their target, but can also enable or disable features that we can come up with, without the need to dig through the source code or forums to find the relevant code. It should be a lot easier to maintain the code that way, and as soon as someone implements a new target (such as the Xilinx), it should be easy to add that target, and hopefully make it possible to use the optional features right away.
I hope everyone agrees this is a good idea, and I invite everyone to join, once I get going: instead of posting ZIP files with your version of the P1V with your cool features, you would simply fork my repo (or even better: the Parallax repo once that gets set up) and post a Pull Request along with your forum post. I also think we should start a Wiki and/or web page on Github to keep track of features and introduce them to newbies quickly.
Thanks!
===Jac
I've read most of the threads in this forum and I'm happy to see there are so many good ideas about possible improvements to the P1V. I would like to see all those ideas in a single place, so here's what I've been up to.
I created my own Github fork (https://github.com/jacgoudsmit/P8X32A_Emulation) based on Mindrobots' fork because he added the BeMicro CV tree to Heater's repo, and the BeMicro is the board that I ordered to play with this (remaining stock at Arrow is now down to 3 so get them while you can, hehe). I noticed there was a large number of intermediate files and output files in the repo, which shouldn't be there (they can always be re-generated by doing a build) so I cleaned things up and entered a pull request to MindRobots.
I also noticed there's a lot of redundancy: all Verilog files are repeated in each of the three target directories, and they are mostly identical. That's a problem because if someone would want to implement some cool feature in one of the Verilog files, they would have to remember to change the files in each and every directory. This doesn't agree with the DRY principle (Don't Repeat Yourself) and since my board isn't here yet, I figured I might as well spend my time on taking care of fixing that. I'll probably just make three (for now) top files with different names, all using a single copy of the Verilog files.
I'm also thinking of creating a simple system to conditionally compile features such as port B, larger hub memory, character ROM etc. That way someone in the future can simply download one repository and build it into the standard P1V for their target, but can also enable or disable features that we can come up with, without the need to dig through the source code or forums to find the relevant code. It should be a lot easier to maintain the code that way, and as soon as someone implements a new target (such as the Xilinx), it should be easy to add that target, and hopefully make it possible to use the optional features right away.
I hope everyone agrees this is a good idea, and I invite everyone to join, once I get going: instead of posting ZIP files with your version of the P1V with your cool features, you would simply fork my repo (or even better: the Parallax repo once that gets set up) and post a Pull Request along with your forum post. I also think we should start a Wiki and/or web page on Github to keep track of features and introduce them to newbies quickly.
Thanks!
===Jac
Comments
I'll certainly honor your pull request.
Removing the duplicates and build output files will be a big help.
I need to make sure my code is up to date. I had stopped doing things with GitHub until Parallax did something but I will support your work since you are moving forward.
I think we will need to figure out how to absorb code from those that don't want to play in Github. I can understand their feelings and we sure don't want to miss out on their contributions.
Is the use of branches the best way to capture individual features? Portable branch? Big hub branch? No video branch? At least until conditional builds are understood in Verilog.
Lots of admin work to go on behind the coding!!
We now have a fork or a fork of the copy of the original release. This cleaning up is making a bigger mess:)
I believe the BeMicro changes are coming from Chip so if they don't break the original target builds I would expect a pull request from mindrobots to get the code into my repo which is the upstream source of his.
Similarly for any clean up that is going on.
Chip has hinted that he make take my repo into Parallax ownership. Might happen, might not. But if we are going to do this and maintain a consistent repo history things should flow upwards into it.
Just now I'm thinking that a "BeMicro" branch is the way to go. As you say when someone figures out how to do the conditional and multi-target builds in verilog then is the time for them to merge any changes in different platform branches together. Same would apply to a Xilinx version or whatever.
I am hoping Chip makes a decision on this take over/or not soon as I will not have time or the knowledge to be a maintainer.
I've pulled mine and pushed the pull to you so you can pull yours (or however you say that in Github talk without getting censored). I had to check to make sure the reset fixes were in the source (they were) before I sent it up to you.
mindrobots, I see you have made a pull request to zicog/P8X32A_Emulation. It is titled "Cleanup work done by Jac Goudsmit" but does not mention BE Micro changes.
Edit: OK I looked into it and sure you have BE micro changes there as well.
Unless you have a better suggestion this is what I will do:
0) Have a quick look over the changes that show up in the pull request.
1) Build the P1 for my DE0 nano from your repo and see if it still builds and runs at all.
2) If so I will hit the merge button and hope for the best.
This will have to wait till later today.
Stupid Git question:
After doing the pull from Jac to Me on Github, I want to get the repo on my PC to match. I thought I could do a simple pull from my PC but that starts to complain and then aborts. Is there a "correct" way to bring my local repo into sync with the dramatically modified repo I have on GitHub??
You seem to have made changes in the repo on your local machine. Whilst at the same time making changes on your github repo.
Things can get very confusing at this point and we enter "git twighlight zone" at least for me.
I guess the correct solution depends on what you want to do. If you are happy to throw away those local changes and get a clean copy of your git hub repo. Or do you want to keep those local changes?
In the extreme I would just delete the whole local repo and git clone your github repo again. That would do it!
Or you can do what is suggested here on stackoverflow http://stackoverflow.com/questions/1125968/force-git-to-overwrite-local-files-on-pull
If you want to keep your local changes things get a bit fuzzy for me.
Are those changes committed yet?
How about "git stash" as discussed here: http://stackoverflow.com/questions/14318234/ignoring-your-local-changes-to-the-following-files-would-be-overwritten-by-merg
Edit: Or this might be enlightening: http://www.szakmeister.net/blog/2011/oct/12/reverting-changes-git/
Edit: Looks to me like those files git is complaining about are derived files not source files. I guess git wants to delete them, they should have been removed as part of the clean up, but is polite enough not to.
I'd just blow it away and start again!
As for the poorly documented pull, I will send you a new one with all the changes documented so the repo history is correct.
I start to worry. Rewriting history in a git repo is not so easy. In deed that is the whole point, to keep the history.
I guess though if you move your repo forward and make another pull request with perhaps a clear title I can take that one and ignore the first.
In normal situations, whenever you have changes in your local copy that are not checked in and the remote copy has has new check-ins, you would need to do a merge. This is more likely to happen as more people join the project, but it's part of daily life when you work with source control.
In this case, it shouldn't be a big deal. If you really had changes in your files that you still want to check in later, you can do a "git stash --include-untracked" (without the quotes) to "park those changes", then do a pull, then do a "git stash pop" or "git stash apply" to re-integrate the changes. Note that the changes I made are mostly deleting files that are supposed to be temporary, so what you could do is undo the changes to all those files and you'll see what else is left.
If you didn't make any changes to the source files or you don't care about them, it's probably easier to just clone your repo to your local drive again (either in a different location, or in the same location after deleting all current files there (including the hidden .git directory).
===Jac
DISCLAIMER: I have 10+ years of experience with source control management (about 15 years usage experience), but I've only used Git and Github for about 2 years. So I'm very familiar with collaboration issues like this one, but I may not know enough details at this time to e,g, give you the exact commands to fix things ;-).
Multiple branches are useful because they make it possible to work on a feature without interfering with other features. Forks are basically identical to branches: Github.com invented them to differentiate between branches that are owned by a single user vs. branches that were made from one user by a different user.
I agree that features can and should be developed in multiple branches (I will create a separate branch to do further cleanup in my own repo), and I could argue that the three (originally two) directories for the various target devices should be (or should have been) branches, not directories. This would be the expected way to store multiple instances of the same file, each used for a different purpose.
But my experience is that in the long run, more branches means more pain. If each target would have its own branch, each added feature means that the feature has to be integrated to other branches (just like changes that have to be copied between directories in the original ZIP files). This is only practical if the project has a source control manager with a lot of time to do this kind of merging all the time (especially in the situation we are in: lots of people who are willing to contribute features).
Keeping the original contents of the Parallax ZIP files in one branch, and keeping separate branches for new stuff including my cleanups, is a good idea, but could be confusing to newbies. When a newbie wants to try things out, they want to have a single place where they can get what they need, even if they don't know what they need. If Parallax decides to start a Github account, it would be the most obvious place for newbies to go: it would be the "master's master branch", so to say. But if that branch would just contain the code from the original ZIP, this wouldn't be very inviting for newbies to engage: they'll just think "okay this works, now which branch (out of potentially dozens) do I have to switch to, to get feature XYZ?".
It would be better if the "master's master branch" would be were the latest (stable) features are stored, in an ongoing process of improvements. Remember, Github makes it possible to label things as releases, so if we would work this way, it's always possible to find the label "Original 2014-08-06 release".
While Git and Github encourage many people to participate in a project, it's easier for people to think in a linear way. It's easier for newbies to read a CHANGELIST.TXT that lists when each feature was integrated into "the main release". Preferably, that "main release" would be under Parallax control of course, mainly to give it some sort of authority. But really, the users ultimately decide what "the main branch" is: anyone can fork my Github repo or anyone else's repo. I expect that the one that's most useful and accepts the most pull requests will end up being "the main branch". If that one is more-or-less officially under Parallax control, it would help to give newbies the "warm and fuzzies" about how well this will work.
I would be happy to volunteer to maintain the P1V repository for Parallax, if they would ask me. As I mentioned, I have lots of experience maintaining source control databases, and I don't think it's much work to maintain a Github repo.
Ken asked me before to contribute documentation for PropGCC and I said I would try, but I never got to it, mostly because (1) I really don't like Google Code, (2) because I couldn't think of a good subject to write about, and (3) because writing and maintaining that kind of documentation would take a lot more time than reading this forum and handling pull requests, once a day or so. Maintaining a Github repo would be a much better defined task than writing documentation so it's much easier to define a task as "done" or "not done".
===Jac
1) saved the two pin map files that were different outside of GIT's control
2) saved my P8X64 directory away from GIT
3) blew away my copy of the P8X32_Emulation repo locally
4) Cloned it again
5) added the two pin mapping files back into git in the DE0_Nano directory
6) committed those to the GitHub repo
7) edited my Pull request to Heater to reflect all the changes in the title (and presumably sent it back to him or it was still in Q for him)
8) pondered further actions......
OK, great, I will be on that later.
I'll volunteer to help! I have no experience doing this and can help find all the ways to bugger it up from a new user's perspective!
I'm seeing how this can get confusing and uninviting to a new user (especially a non-Github user).
Right now, we have:
Heater's Repo (master branch) | |---------> mindrobot's Repo (master branch, planning on adding branches soon for features) | |---------------- Jac's Repo (master branch)1) Heater did initial formatting and gathering into Github
2) I forked Heater and added the BE micro directory and the reset bug fixes (in my local repo, I created a new directory for P8X64 but we'll circle back to this)
3) Jac forked me and did the clean up work and when happy with his work, sent me a pull request
4) I pull/merge from Jac so at this point we match.
5) I sent pull request to Heater - then added two more files to the DE0_Nano directory (pin maps)
At this point, (until Heater does his merge from my pull request) none of us match!
When heater does his merge (triggered from my pull request), Heater and I will match. As soon as Jac pulls from me (to pick up the two new files), we will all match again.
Now, if I understand this correctly, it is up to Jac (or any downstream repo owner - someone who forked) to check with the parent to see if they've added any updates and if there are updates, then how do I make my repo I forked from Heater match his repo? I don't see a way to request a pull from my repos parent. (Am I missing some basic concept here?)
As an example, I'm going to create a branch in my repo for my PortB P8X64 code. I will do that locally on my PC, work on it until satisfied with it and then push it as a branch to my repo.
At this point, I can send Heater a Pull request and he can decide if he wants the branch as a branch to his Repo.
I could take the branch and merge it into my master branch?
Jac comes along and see it and decides he want the code in his Repo......and it's back to the question of what does he do when he wants to pick up an upstream change?
Am I just confusing me at this point?
Just like with the original fork, you don't need permission to sync your repo to its parent and you don't have to put in any kind of request to do so. However it does take a few commands: see https://help.github.com/articles/syncing-a-fork. Merging the parent to your branch is pretty straightforward and should be automatic unless you made changes to your branch too: then you'll have to resolve the differences.
I just synched my repo with your changes by using the three commands from the page linked above, and it works: my history now has your latest change "Add pin mapping spreadsheet..." in it.
I don't like how it looks on the Github web page (https://github.com/jacgoudsmit/P8X32A_Emulation/network) because at first glance it appears as if I made the change myself. However, when you click on the latest dot in my timeline, the history correctly shows you as the one who made the changes, and that's what it shows everywhere else too. So I think the problem that there's no arrow in the graphic as you might expect, is just an issue with the Github web code that generates the graphic. I'm not going to worry about it for now: it's obviously stored correctly.
===Jac
PS Now that I'm getting a little more "intimate" with git and github, I think I'm going to have to look for a git tool with a GUI, other than the Github for Windows program which is fine for the basic operations but can't do almost anything we talked so far. I'll do some research later today. If anyone has any good suggestions, let me know!
EDIT: I think I found "the one": Git Extensions (https://github.com/gitextensions/gitextensions)
It has tons of features (looks like you don't have to use the command line for anything), and appears to be well-organized. My favorite feature is that it looks mostly like a regular old-fashioned Windows program; no stupid skins or annoying frameworks that everyone seems to want to put on their software nowadays. It allows you to use two of my favorite tools for diffing files: P4Merge and BeyondCompare (and many others, the default is K3Diff), and it even allows you to start other Git clients on your system such as Git Bash, Git GUI and GitK. Recommended!
See this new thread for more information.
===Jac