Documenting/organizing/keeping projects
Ale
Posts: 2,363
Hei all,
I'd like to gather a couple of ideas on how do you cope with documenting projects. I have like you many projects: software-only, sw-hw, only hw...
There are sw, datasheets, schematics, drawings on paper, models.... I'm just overwhelmed by the amount and diversity. I thought I should have a wiki server (there are plenty!) and document there with links to datasheets, drawings, schematics, all what is needed. Kind of replacing a paper notebook. I have several paper notebooks, there are projects and ideas and all mixed.
I started with a projects directory where I have one folder per project (at least sw projects, hw too). But written docs are not to be found, and not written, datasheets are somewhere else all together grouped by manufacturer or kind like memory and displays and parallax, to name a few.
You see the problem, dilemma... how do you cope (or try to) ?
I'd like to gather a couple of ideas on how do you cope with documenting projects. I have like you many projects: software-only, sw-hw, only hw...
There are sw, datasheets, schematics, drawings on paper, models.... I'm just overwhelmed by the amount and diversity. I thought I should have a wiki server (there are plenty!) and document there with links to datasheets, drawings, schematics, all what is needed. Kind of replacing a paper notebook. I have several paper notebooks, there are projects and ideas and all mixed.
I started with a projects directory where I have one folder per project (at least sw projects, hw too). But written docs are not to be found, and not written, datasheets are somewhere else all together grouped by manufacturer or kind like memory and displays and parallax, to name a few.
You see the problem, dilemma... how do you cope (or try to) ?
Comments
The best thing ever for organizing my hobby code was the arrival of Github and bit bucket. Or just using git locally.
I think it's best to keep every data sheet and other related documents in the project directory somewhere. Every project contains all the information it needs.
That might mean you have many copies of the same thing around the place. Never mind, they are not usually so big and storage is cheap now a days.
I read that as "I can keep a straight face when I'm paid to because it is hopeless and for hobby projects too" and rofl!. I know it is hopeless but "continuous improvement" may help
I keep two or so (out of 100) projects in a local github, too
I bought two new paper notebooks last week
-Phil
For daily revisions, research and development (WIP), I create a date folder within a core working folder and copy the core contents into the date folder and continue working from the core folder. I usually only keep a week or two of date folders within the core folder before I purge them.
It has worked well for me, but you have to be diligent and maintain it. Between work, home, and outside contracting, I have about 6 projects right now that otherwise I would not be able to keep aligned.
Some projects are simple, while others can be quite complex. Some projects can be made up from one drawing, from which a prototype is made, while others require several different drawings for various portions of assembly. Then there is the datasheets and notes for the various aspects of any project.
I begin with a main project folder, and for each important aspect of the project, I create a separate folder. I found this to be the easiest method for locating various aspects of a project. Kind of jumping ahead here, but..... handwritten notes and drawings are eventually put into digital format with CAD and a word processor (notepad), with drawings being placed into an appropriate folder and notes being stored within the main project folder for easy access. Please note, that I specifically use text files for project notes, which make them very easy to locate, for all other word processing, I use MS Word. If a project requires software, I usually create a sub-directory in the main project folder and sometimes, if the software is complex, I will create sub-directories just for the software.
I keep all of my datasheets and other PDFs in one giant folder, with many sub-directories, which either specify the subject matter or manufacturer, and occasionally I will create copies in one or more folders, to make it easier to find. If necessary, I will create shortcut links to these PDF documents in my project folders.
As for hardware (prototypes in my case), either they are junk or they have a valid purpose to exist. If they are junk, I salvage usable components for future or current projects as needed, but if they have purpose, then they are occasionally smiled upon
Great question, Ale! I recently found weaknesses in the process that I use at home, so I will not confuse anyone with that mess. However, I admit that I have many printed data sheets with written notes and "cheat sheets" for the most relevant info.
At work, I tend to use Microsoft OneNote - its tabular structure better matches my thinking process. From there, I create hyperlinks (like Beau) and put handwritten information into digital format (like idbruce). If handwritten notes can be summarized cleanly, then I will recreate the content (bullet points, table, etc.). However, sometimes there is an elegance to the stream of thought that went into writing notes, and I just scan them into PDF or picture form.
I have now used OneNote at 3 different employers, and each time required vastly different structures. At home, though, I just cannot seem to emulate the same degree of efficacy.
Regarding Phil's comment about versioning - it really needs to be embraced fully or abandoned fully. Our company is experiencing growing pains, and amongst them are revision control. Supporting warranty claims requires tracking compatibility - electrical and mechanical. Can I swap out a "Rev F" board with a "Rev G" board? Is that consistent with our regulatory files? Supporting new orders while burning through old inventory also requires impeccable revision documentation. Doing it half way and filling the gaps with tribal knowledge is either wildly inefficient or a disaster waiting to happen.
I like notebooks, paper ones, I can put my ideas down about a particular project, I just jump to another project the day after and the next page has no relation to the previous one... o well... I originally bought an iPad with the wild idea of keeping it as my notebook... yeah, drawing with the finger ? yeah, that _is_ a joke. The "drawing" software was just not there... I got a Samsung Note 10 with the stylus, that works better but I just cannot get accustomed to using it, but it is far better than what the iPad used to (not) provide. No idea about the iPad Pro or whatever is called, I don't sink any more money for Apfel products. (Not that I ever had many of their devices).
I see that no one went for a wiki but hatallica went for OneNote. I just don't want any MS software. Enough that I have to use w10. But all seem to have a kind of blackboard of sorts with datasheets and drawings as "post-its" .
Thanks for the input. I'll have to think a bit more about the whole process
They are portable. My note 4 phone has a pen, and I've gotten to where I can use it nicely. Same phone takes photos of stuff, and that leaves me with a bitmap, text files, maybe excel at times.
One really great bitmap tool is "ad-hoc datasheets" I'll be wanting to know about something, or get some info organized. Open up a big bitmap and just start pasting whatever it is I want in there. In the mechanical space, I'll do this frequently for tooling specifications, fixtures and a lot of other things that need documentation. Couple of pictures later, and it's done, easy, obvious, useful.
I don't mind Microsoft Office, and will use it. I got started with bitmaps and text quite some time ago on IRIX. It was a pretty great flow, and I just kept with it, and all my stuff more or less works on anything I put it on. There isn't much that can't render text and pictures.
I start with a project name (can be a code). I append "-000a" to start with.
As I proceed, I do a save-as to increment the alpha. When it works or I get to a good point, I increment the 000x to 001a. I backup to external devices (now hdd) all files regularly. I also zip including the compiler if necessary at strategic points.
I have not lost anything in over 40 years of programming and hardware design. I have been able to go back months when a project broke because an editor truncated a file and I didn't realise this for some time (many backups later).
As for fixed documents, I have a directory for pdfs such as chip specs etc. I just ensure the filename is meaningful, such as "EEPROM 24C512 ...... .pdf"
I have no need for git. It is easy for me. I always document code, and include lots of change notes at the top of the file with a date and initials and version. KISS
And, it also provides a proof of authorship should it ever be required.
If I have the option to compile without saving, I will open one, make changes, and if I'm happy about those, save as to the next increment. Otherwise, I'll save as the next increment, then make changes.
The idea being to preserve full project states.
Dependent files get the same treatment, and what I end up with is a buildable, running history of what I"ve done to date.
This does tend to accumulate files. I don't care. Disks are cheap, and I've gone back through the history more than once to follow an idea that I should have, but didn't know it. Just one of many things.
Sometimes, this workflow does require touching a few, or all files, where things are included in other things. Doesn't take long, and it's really nice to have the ability to run, "the one I thought looked good last week..." or just go back a step, whatever.
If I arrive at "done" or something to release in general, I will often pluck that part of the tree out and tidy it up some, but not always. Depends on the nature of what it is.
If I am using a version system, it's generally the same flow, with me making sure to trigger a refresh of the data in the system frequently, and at key times to be tagged as a version that will build.
To me, this kind of thing is essential. Often, I'm learning new things on the go, or remembering things I thought I knew, or had wrong. So it's little steps, test this, build that, prove this thing over here.... Without some history, it's very difficult to make things incrementally.
You've never accidentally broken something without noticing it, only to try using that feature months later to find it's broken? Version control can make fixing these kinds of bugs so much easier - it gives you an old working version, and you can tell your VCS to show you all the differences between the working and current versions, so you have a better idea of where to look. It also acts as a sort of backup, in case your editor goes haywire or something.
Git has a bisect feature, where you tell it a known good version and a known bad version and it does a binary search, presenting you with different versions and letting you test it and tell it if that version is good or bad. You can even automate this, by giving git bisect a script that automatically tests each version that Git gives it and returns a value indicating if that version is good or bad. This lets you narrow down when bugs like this snuck in to a single commit.
The net was nice progress.
Evernote and Dropbox are my other go-to tools. Take a picture out on the shop floor, and it's waiting for me at my desk by the time I walk back up.
EDIT: Sorry, meant to say git integration, not specifically Github.
For the time being I'll extend my current methodology with a couple of tips from you like duplicating the datasheets and keeping revisions of full directories, space is cheap these days
Something like versions of folders provided by win10 would be an idea, I just don't want to lock myself into it, ms is know for dropping their "solutions" and the drop of a hat (pun intended).... And returning to GNU/Linux is what I have in mind....
Thanks a lot !
once upon a time I played around with 'wiki on a stick': http://home.arcor.de/lustiger.harry/ for documentation. But github contains all that functionility and more. Nowadays I run gogs (https://gogs.io/) @home, running on my home server.
And when my webserver was still up and running it was also based on git, with hooks to translate markdown documents into static html files whenever I committed them.