When you use the child hair in poser. It will look like a white wire mesh, but when you render her, the hair will look like it is suppose to. It is just something poser does to DAZ opjects.

extracting Source characters and textures: a how-to guide

What this covers


This guide will  go through the process of getting files for a Source SDK character that can be edited in a modeling program and an image editing program. The process for non-character objects would be similar.
 
You will need

VTFEdit
GCFScape
MDLDecompiler (drop this .exe into C:\Program Files\Steam\steamapps\\sourcesdk\bin\ep1\bin


How it's done



Extracting from GCFs
First, a little background information. All of the Steam game assets are found in GCF (Game Cache Files) files that relate to each of the games. They're all packaged together like that so that the game assets aren't changed when you're working with your own files. You have to extract out your own copy to edit and use.

1) Install GCFscape and in the installer choose to have it associated with GCF files.

2) Navigate to where the GCF files are stored.On my computer (and probably yours too) it's on C:\Program Files\Steam\steamapps.

3) We have to open up two different GCF files to get what we need: one for the .mdl files, which hold the character's structure and animations, and one for the .vta files, which hold the character's texture information. First, we look for the .mdl file. In my case, I'm exporting the female_02 model. So, double-click on "source models.gcf" and GCFscape will come up with an explorer pane. You can find all kinds of assets in there, but I'm going to navigate to hl2/models/humans/Group02 to get my model. Over in the right pane, select all the items prefixed with Female_02, right click, and select "Extract".

4) It'll ask you where you'd like to put it. Pick a folder. I made one on my desktop.

5) Now, we're going to repeat nearly the same process, except for the textures. Open "source materials.gcf", go to hl2/materials/Models/Humans/Female/Group02. Grab and export all the files in that folder to the same folder you put the other stuff in. Those are only the faces for all the females, so you'll also need to get their clothes, eyes, and mouths. These are found a folder up in hl2/materials/Models/Humans/Female/. Select all the files in that folder, and export them as well.

You're done extracting from the GCF file!

Decompiling the model
Now, we want to decompile the model from the .mdl file into some .smd files so we can import them into XSI Mod Tool, or Maya, or whatever other modeling program we're using.

1) Open up MDLDecompiler. It's annoying because you can't browse to the model; you have to copy and paste its location. So, find it in Windows Explorer, copy and paste the address bar, plus the name of the file. Then, you can browse to find the output directory. Make sure all three of the lower checkboxes are unchecked.

2) Click the "Extract" button. If all goes well, a couple dialog boxes will tell you that you've loaded the model, and then that you've completed dumping the model. You're done! If you look in the folder that you dumped it to, you'll find a bunch of .smd files, which are animations and models. You'll find a .qc file, which is code written in Quake C that defines the model and its animations. Then, you'll find one or more .vta files,which define facial animations. We're not opening any of those files yet.

You can edit the SMD files in Maya, given the correct plugins. I'm waiting on a Maya license at the moment so as soon as I get that I'll write a how-to guide on editing SMD models there.

Converting the textures
Now, to get the textures into a format we can work with. The compiler will want .tga files, so we want to decompile the .vtf textures into .tgas.

1) Open up VTFEdit and navigate to the folder where you put all your textures. Or, if you associated your files with VTFEdit at install time, you can double-click on them to open them up.

If you click on a .vmt (Valve MaTerial) file, you'll get a text file that looks something like this:

(for dark_eyeball_l.vmt)

"eyes"
{
    "$basetexture" "models/Humans/Female/dark_eyeball_l"
    "$iris" "models/Humans/Female/pupil_l"
    "$halflambert" 1
}

Don't worry aobut those for the moment. If you open up a .vtf file you'll see an image.

There are bunches of check boxes and stuff on the left that we're not going to worry about for now. They'll be useful on recompile but right now we're just worried about exporting.

2) Go to File>Export and then pick where to save your .tga file. Your file will be exported into that location, then you can edit the Targa (.tga) file using an image editor like GIMP. It's just like editing any other image.

That's it! Let me know if you have any questions.

BMW motorola Symbol

The representative from Motorola contacted me and I found out that the device that they currently use only has audio output, there isn't a type of Video output. The representative said that he was pretty sure that there was not an available device that would do what we needed it to.

Useful Links for Character Creation

Here are some links that I have found that have some pretty good stuff on it.
//both of these have freebies!!
http://faerie-dreams.com/page_3.html
http://www10.sharecg.com/search/index.php?q=maddie&button.x=0&button.y=0

sharecg is where I found the free dress and the free morphs for the hair.
faerie-dreams has the free play clothes, and whimsical costumes and hair props.
Still trying to figure out how to make them all fit.
Will continue to look for more free stuff.

When downloading free stuff, make sure to put them into the correct folders
c->program files->smith micro->poser pro->downloads->runtime->..........

resource for import and modification of Source assets

Stumbled across Kathar's website today. He has a whole section on HL2 modding, specificaly on asset modification (i.e. decompiling and recompiling HL models, which is also helpful for importing in our own models.

yay progress.

tools of the trade: Cannonfodder's model decompiler

To access and modify models, you need to use this decompiler:

http://www.chaosincarnate.net/cannonfodder/cftools.htm

For the executable to work, it'll need to be in:

C:\Program Files\Steam\steamapps\\sourcesdk\bin\ep1\bin


then you can pick which file to decompile.  You'll get a bunch of files when you do:

.smd files are models, animations, and textures, all put together
.qc is a file in Quake C, that instructs the compiler on how to interpret things
.vta is a texture file

Extra steps are needed to get into each of these files. .smd files can be imported into Mod Tool by using the ValveSource plugin, as can .vta files. .qc files are plaintext so you can edit them.

tools of the trade: GCFScape

As I was going through the models in Face Poser and Model Viewer in the Source SDK, I wondered where all those models were stored, because I was thinking about looking at the construction of a couple of them to get some ideas about how to build our own. Model files in Source have a .MDL extension, so I did a system search for ".mdl", but no results! Strange.

I began poking through the folders and found these large files with a .gcf extension. After looking at the wiki, I found out that these are where all the resources for the games are stored. To get into these files to get out resources, you need a tool.

This is where GCFScape comes in. You can download it here:

http://nemesis.thewavelength.net/index.php?p=26

So, you use this little program to extract out all the files used in the game. For example, I'm considering using one of the Half Life woman models for our patient's mom. I opened "source 2007 shared models.gcf", looked in hl2/models/humans/Group02 and grabbed all the files beginning in Female_02, and now I have all her data. I'll have to decompile using another program to really get to the inside of her, but that's a start!

So, when you see these mysterious figures from nowhere in your programs, now you know that they are contained in the gcf file and you can get to them if you want to :)

version control: a quick tutorial

About how it's set up:
I set up version control with Nell over the past few days to manage our files and revisions since several of us will be working on this project and trying to synchronize versions is generally a mess. Additionally, it will track changes for us, which proves to be invaluable on any large project.

I chose Subversion for our version control because I'm familiar with it and it's well-respected. Since we're using a Windows Server, I chose svn1click for easy setup and start up. It does most configuration for you through a Windows installer and goes ahead and starts the Windows service so you can access it.

I modified permissions to allow for all of us to access it through username and password, and to disallow anyone else from accessing it. Here's a little guide on setting a few of those things up, for future reference. It's super easy.

About how you should use it:
To access the repository, you need to download a svn client. I (and a lot of people) use TortoiseSVN. It integrates right into your right-click menu, making the repository easy to access. Go ahead and download and install it.

To set up your repository for the first time, make a folder where you'd like to keep your project. Open the folder, then right-click on the inside of it. You should see an option for SVN Checkout. Click it.

You'll get a dialogue box. Fill it out with the url of the repository, just prefixed with "svn://". In our case, that's svn://vegroup.cs.clemson.edu. The second folder should be the folder you just right-clicked in. The other settings are fine as-is.

 

After you click "Ok", you'll be asked for username and password. Ask me, and I'll tell you what yours is--it's different from your other passwords! After you authenticate, you'll get a progress dialogue box, and then if you look in your folder, you'll see stuff has been put there. The green check beside the folder means it's under version control. 

If you open up that folder, you'll see three others: branches, tags, and trunk. For now, we won't be touching "Branches" or "Tags", only "Trunk". Trunk is where all your work should stay, for the most part.

I'll step you through four processes right now: adding a file, modifying a file, updating your project, and restoring to an older revision.

Adding a file

 Open up the trunk folder. Make a file (any file will do for now). I'm going to make a text file and type some random gibberish in it. Now, let's say I want to add this to the project. Right-click in the folder, then choose SVN Commit.

When you click, a dialogue box will come up. In the top text box, write a comment that describes what you're adding. Comments help your team understand what you're doing, and find revisions when they  need to. The bottom pane is a list of files that you've changed. You can see that beside my new text file it say "non-versioned", meaning I've never uploaded it before. I am going to check that box so that it is uploaded.

I enter my login information and click okay, and now my file is under version control! Yay!

Changing a file
To change a file and its contents in the repository, just open up the file, do what you need to in it, then follow the same process as above.  Choose SVN Commit, then follow through the dialogue boxes. Your change is made!

Updating your project
Let's say that someone else was working on the project and added or changed some files too. It's good practice to make sure your files are up to date before you start working on them. So, before you sit down to modify any piece of the project, right-click and choose SVN Update. After clicking through a few dialogue boxes, your project will be brought up to date with whatever's on the server.

If you think it through a little, it's easy to see that this can cause some problems. What if you and someone else were changing a file at the same time? SVN handles this nicely. Instead of just overwriting your file when you update, it will show you that the files have a conflict and give you options on how to merge them. That's for learning about another day, though. Another thing you want to think about is that other people are using the code you check in. So, try not to check in anything that is broken, or else, other people will try to use it and be frustrated. Only check in once you have useful code. There are ways to get around this (like creating branches) but again, I'm not covering that right now.

Going back a revision
Let's say that I made a change in my code that turned out to make things go wrong, or that I realized I liked an old version of something better. SVN keeps track of all revisions so that you can go back to old versions.

To do this, right-click on the file you want to shift back. Then go to TortoiseSVN>Show Log.

You'll get a nice little list of all the revisions that have happened, along with the comments (this is why comments are important!!).

Then, you have bunches of options. "Compare with local copy" will show you the changes since that revision. "Save Revision to" will let you save that revision to another file. Open/open with will, well, let you open it. You can also pick "Revert changes from this revision" to make the changes from that revision go away, or "Update item to revision" will change your local copy to that revision.

Okay, there's a bit to get you started! You can do a lot more complicated things with SVN. Read about it yourself, or I'm sure I'll be posting more detailed steps for common tasks later.

starting point for character animation

In reading, it's become clear that any body animations associated with a character must be exported inside the SMD file for that character. There's not good documentation for how to export, though. The closest I can come to it is to find how SMD files are formatted:


http://developer.valvesoftware.com/wiki/SMD

Under that "vertex animation" section, that's where our stuff would be, but as you can see, that section is kind of undocumented. I'll spend the next few days trying things and looking at the file to see what I can decipher, as well as asking as many people as I can think of about how to do this.

Structure of a Source Mod

There are good step-by-step guides for setting up a Source mod, but I want to take a little bit to explore the structure of a Source mod after it is set up in its folders. Our team will be using version control and working from the same (already set up) project, so this is important so that we're all on the same page.

When you create a mod, it creates two folders: one at a location you specified, and one under the Steam application path. So, for my working copy, these folders are:

C:\virtual_patient
C:\Program Files\Steam\steamapps\SourceMods\virtualpatient

Let's talk about what's in each of these folders.

C:\virtual_patient
Here's a screenshot of its layout, partially expanded. (Yes, I misspelled "patient", haha.)

The folders are fairly well named. We don't really have to deal with most of them. The ones of interest are:
mapsrc: where uncompiled maps made using Hammer Level Editor should be stored
materialsrc: where textures and materials that we use for our game should be stored
modelsrc: where uncompiled models files should be stored (i.e., SMD format).

Of course, we are also concerned with the src folder, but only a small portion of it. Opening up that folder yields a bunch of subfolders holding source files, as well as two Visual Studio 2005 projects (don't worry, you can convert it to 2008 as well). We're not too concerned with the Everything_SDK-2005 solution. The Game-Episodic-2005 solution is the one that all our work will be done in. Opening that project will show that it's associated with a LOT of source files. This solution contains two projects: Client Episodic and Server Episodic. We'll only be working in Client Episodic.

If you open that project you'll see bunches of source files. Again, most of these will not be important for our purposes, since we're not shooting guns, fighting monsters, climbing ropes, or anything like that. What you're actually looking at is the full source code for whatever game you're modding. As you can imagine, our mod will probably end up deleting or not using the vast majority of these files. The ones that will be important to us are the more utilitarian ones, like the ones for physics, camera movement, level rendering, and human behaviors. However, this gives us some GREAT source code to work from. You can wade through a few files and try to make sense of them if you'd like.

C:\Program Files\Steam\steamapps\SourceMods\virtualpatient

All right, on to the second folder. In the root folder, there's gameinfo.txt, which holds important information about our game and might have to be modified eventually. The rest of the files you can kind of ignore in there, I believe. The folders here are well-named, too.

The cfg folder holds .cfg files for each "chapter" of our game (in our case, might end up being different nurse interview scenarios, for example), along iwth config.cfg, which looks pretty important, but I'm not sure if we'll have to modify that or not. Under Expressions you can see pictures of animations and phonemes for some characters; not sure why they're there at the moment, except that I opened them in FacePoser and perhaps it placed the images into my project. Under Maps are maps that you compile to run in your level using the Hammer Editor. In Scripts and its subfolders there are various animation scripts for vehicles, people actions, and effects. Of course, that will be very important for us in our project.

That's a brief overview of the structure of the mod that Steam sets up. More about the location of utility programs and files later!

Step By Step: Exporting a Character from Poser to Half-Life Uncompiled Format

Since there is not a good step-by-step guide, I am going to write in detail how to get a character out of Poser into a format that Half Life can use, with instructions and screenshots.

This process was taken from the two YouTube videos cited in the previous post.

Tools You'll Need:
Poser (another modeling package would probably be similar)
Source SDK
Mod Tool 6.01
ValveSource Plugin for Mod Tool

Exporting from Poser

1) Open up Poser Pro (or your other program) and get the model you want. I'm using Simon because he's dressed and ready to go. Select File>Export>COLLADA.
EDIT: In order to make the textures map correctly, try exporting as a .obj file and importing that into Mod Tool. Sometimes COLLADA doesn't work right.

 

2) In the next dialogue box, select Single Frame export. I haven't figured out animation yet, but it may be that in the future multi-frame export is useful for animation. Separate post for that!

 

3)  In the next dialogue, make sure that only Simon is selected.

4) Give your file a name in the following dialogue box. In the one after that, change the scale to 1000%. Uncheck "Include Normals" and "Include Rigging" since we'll be rigging the character ourselves. Everything else should be set up okay from there. If a "Select Morphs" box comes up after that, just click "OK".

Importing into Mod Tool

5) Yay, your model is exported! Now, open up Mod Tool. (BTW, all these programs take up a LOT of memory so if you run them at the same time, your computer might crash. I advise opening each separately.)
In Mod Tool, go to File>Import>Model...

 

6) Choose Collada (.dae) on the File Type dropdown in the Open dialogue. Find your model and double click it to import. You'll probably get an error in red at the bottom: Triangle Count Exceeded. Don't worry about it.

Reduce Polygons
7) Let's go ahead and reduce down the polygon count to make it easier to work with (and allow for export into Half Life). Choose Model>Poly. Mesh>Polygon Reduction.

8) When the dialogue box opens, use the "Units" drop down menu to select "Final triangle count" and then beside "Tri Count" type "7000". Half Life will allow for up to 7500 polygons for its characters, so I'm just exporting a little below that count to be safe. Immediately your image will change to have fewer polygons. You'll see a visible difference.

Exit out of that dialogue box.

Freezing Transforms


9) To make your model export correctly, you need to freeze all of its transforms. This is accomplished by clicking "Transform" in the right hand menu then clicking "Freeze All Transforms".

Baking the Textures

10)One more step before we line up the skeleton: exporting the texture map. When you exported the COLLADA file, in the same location, all the textures were saved with it. Although they don't show up on your model right now, they're still associated with the appropriate parts. What you need to do is to let the program make one texture out of all those textures and to save it somewhere. To do this, click on the character to select it, then go to Model>Property>Render Map.

11) When that dialogue comes up, set X Res to 512 or 1024, whichever works, then next to UV, select New>Unique UVs (Polymesh).

12) Set the Path to wherever you want to put it, preferably in the Source SDK material folder that you'll be using. Choose Targa(.tga) under "Format", and under the first dropdown under the Map section, select "Surface color only (albedo)". Then, click "Regenrate Maps..." and exit out of the dialogue box.

 

13)  Press 8 to open the explorer (handy shortcut!) and find your object. Delete any materials that are already there.

14) On the right sidebar, under "Create", click "Textures", then click "Image."A dialogue box will show up. 

15) Click "New" beside the Image dropdown, and find your image that you saved for the texture.Yay, your model is textured! If you choose "Textured" under the drop-down view mode menu to the right of the screen, you'll be able to see your texture. (You can see my texture didn't turn out so well in this screenshot. I'm not going to take the time to fix it now because usually it turns out okay. Look for a separate post on texture tweaking.

 

Add & Align Biped Guide

16) Now that our character is all set up, we're ready to put bones into it! Choose ValveSource>Character Rigs>Biped Guide. (I'm going back to wireframe view for this.)

17) When you do that, a "skeleton" will show up on the screen with different lines connected by boxes. But it is HUGE! Type "z" to activate the zoom tool, then right-click to zoom way out to see the whole thing. You should see your character in between the legs of the guide.

18)The boxes act like joints and the lines act like bones. Your goal is to line up the bones and joints as closely as you can with the Poser model. So, you'll want to select the model and use the Scale, Rotate, and Translate tools to line up that character. First, uniformly scale up until the shoulders and head kind of line up. You can do this by typing scale coefficients into the "Transform" box near the "S" on the right side of the screen, or by clicking "S" and then dragging the box in the middle of the three axes. I like typing because it's quicker, but sometimes it doesn't work correctly. You also might want to switch from 'User' perspective to 'Front' perspective so you don't have to deal with the distortion that the camera brings while resizing.

 

19) After you get it scaled up appropriately, click on the red boxes to select them ('y' is the shortcut key for getting into select mode). Then, click on the R and T buttons for rotating and translating, and make the bones line up with the body. Other helpful keyboard shortcuts: Zoom tool, press 'z', then middle click is zoom in, and right click is zoom out, and left click + drag is reposition.

  

20) This part is more of an art than a science. You need to line up all those bones from Top, Right, and Front views. The hands will be the most difficult part since you have to move the position of each of those fingers.  I'm not going to take a lot of time to do my alignment since I'm primarily doing this for tutorial purposes, but go ahead and spend some time to make this right. A quick check shows all my bones are decently in line from each perspective.

Rig the Character

 21) Now, select ValveSource>Character Rigs>Rig from Guide. It'll do a bunch of magic.You'll end up with a different looking model and bunches of dialogue boxes. You can exit out of all those.

22) Press 8 again to open an explorer, if you closed it. Click on ValveGuide, then press Ctrl+T for "Select Tree", then press the Delete key to delete the guide.

Envelope the Character
23) Almost there, I promise!   Expand ValveBiped in the explorer. Click to select your character, then go to Animate>Envelope>Set Envelope.

24) Click "Yes" in the dialogue box that comes up next. Then, your mouse cursor will turn into a little arrow with "PICK" right beside it. Now, left-click to select, then right-click to confirm. Sometimes this works weirdly for me but keep trying that process and eventually you'll get a new dialogue box (that you can close) and a colored-point version of your model.

 

Tweaking the Points

25) All right! You're close to done! Use the rotate and translate tools and select some of the random sphere things that were added to your model. Play around with them and make sure that your character is moving fairly naturally. Try nodding/shaking its head, moving arms forward and backwards, etc. If you find something that's deforming strangely, then you'll want to edit their point weights. I will cover that later in a fine-tuning tutorial.

Exporting

26) Finally! Click on ValveSource>Export to .SMD. Pick the location of where you'd like to save the file (probably in the models source folder of your mod). You can leave all the other settings. Click the OK button!

You're done!! Well, kind of.

The next steps are compiling together the model you just exported and its texture into one file that Half Life can handle. I will be covering these steps in a future tutorial, since I haven't worked quite that far yet.

Remaining things to do for character importing:

- Actually getting the character into the mod

- Finding out how to do animations using this model in Face Poser

- Figuring out if there's anything additional you have to do to allow for lip syncing

Final Words

Yeah, the process seems long and overwhelming, but the truth is, after you've done it 10 or 12 times (as  I have while trying to get things right) it's not too bad. You'll get better at it. Any modeling tool has a steep learning curve but after spending a bunch of time using Mod Tool and learning the different shortcuts you'll get much quicker at it.

Please let me know if you have any problems or questions!

solution for exporting characters from XSI into Half Life format

Download a version back of the Mod Tool (6.01):

http://www.moddb.com/downloads/xsi-mod-tool-601

Use the most current plugin:

http://usa.autodesk.com/adsk/servlet/item?siteID=123112&id=13525256&linkID=12544121

That will allow your rigging to work.

These two videos go through step by step on how to accomplish the whole process:

http://www.youtube.com/watch?v=hjsstUqiUyY&feature=PlayList&p=EB45F2480C9A5DA3&index=29
http://www.youtube.com/watch?v=sknm9_CuHps&feature=PlayList&p=EB45F2480C9A5DA3&index=30

In the next couple days I'll be writing up a step by step guide with screenshots for how to do this since the video is quick and kind of blurry. So far I have a rigged model but I have not compiled and put into Half Life yet.

In other news, I got in touch with the person who wrote the C++ functions for dynamic text to speech generation. I haven't taken a look at his code but I'll be doing that soon.

dilemma.

Working with the Mod Tool today to do character export into Half Life, it turns out that the current version of the ValveSource add-on is broken and will not allow the character to be rigged to the skeleton. The earlier version of the add-on works fine, but is only compatible with Mod Tool 4.2 (we're on 7.5 now), which imports barely any file formats, including the proprietary XSI file format that 7.5 would export.

Grr.

starting point for source modding

Here is the link to the Valve Developer Community:

http://developer.valvesoftware.com/wiki/Main_Page

It has pretty good descriptions of most everything you need to get started. Here's some links to beginning topics:

Downloading the SDK
Setting up a new Mod
Installing and Debugging the Source Code: be sure to follow the directions for your version of Visual Studio, or it won't work right. Also, be sure to do release builds and not debug builds. Steam provides a console for debugging, or there are instructions for how to do it through Visual Studio, but I haven't got them to work correctly yet.
Using Source Control, which I will be setting up today so that we can work on the same project. I'll actually probably follow these directions instead because I am more familiar with Subversion. TortoiseSVN is easy to use and it's probably what I'll be using. If you've never used source control before, I promise it will make all of our lives easier, and if you have any questions about it you can come talk to me. I'll do a later post on how to actually use it.

Here's a good starting place for all the other topics for modding. In particular, this week I'll be looking at modeling so that we can get our virtual people into the project. The Softimage Mod Tool will serve as our bridge between Poser Pro and Face Poser, which is Source's tool for character animation. You'll need to download the Source plugin as well to allow for imports and exports.

Modeling and related tasks always have a steep learning curve, but there is a lot of documentation and tutorials out there. For first steps, we'll be exporting our Poser models into the Collada (.dae) format, and importing into the Mod Tool from that format. There are a couple videos about doing this. After exporting into the correct format, the next step will be getting it into Face Poser and figuring out whether we should make our animation keyframes there, or whether we will have to export keyframes from Poser. I'm hoping we'll be able to make all our animations there because exporting and importing keyframes will get tedious.

Enough writing for now. I'm going to set up source control, and I will make a blog post on how to access that project soon.