ArtCraft Developer
  • Content count

  • Joined

  • Last visited

  • Days Won


jtoddcoleman last won the day on February 28

jtoddcoleman had the most liked content!

About jtoddcoleman

Profile Information

  • Gender

Recent Profile Visitors

18,528 profile views
  1. Server Stability Update

    I thought about this last night, and I think perhaps I was too quick to dismiss it as a stop-gap. Even if 5.5 comes together very quickly and we zip it out to Test in record time, there will still be an inevitable delay between Test and Live and I really don't want to suffer through that with such high msg spikes. I'm going to have QA test the functionality again today to see how broken it is.... (does it not work at all? does it work, but it is just poorly messaged?) Depending on how that goes, and how much effort for we expended, I'm willing to entertain it if the cost isn't too expensive. In either case, I want to apologize for being dismissive @Anthrage, you were suggesting a solution -- and whether it works out or not, you were trying to help the project and that should be applauded. Todd
  2. Server Stability Update

    We just added a West Coast tyranny and a Singapore and a Sydney bloodbath. Hopefully that will spread people out and help a bit. Todd
  3. Server Stability Update

    5.5 has some pretty radical architectural changes, so we may see this again or we may not (if the issue is buried inside a system that has since been rewritten, this might be as relevant as a broken window in a building that was torn down -- that is to say, while we didn't actually fix it, it is no longer a problem). If it has anything to do with zoning, persistence or AI, it's almost certain to have been deprecated... and will likely be replaced with a handful of new, equally aggravating issues for us to track down. ;p Todd
  4. Server Stability Update

    ^^ absolutely agree, I'm constantly walking through the office and I see someone playing and there is just enough gameplay that it is easy to imagine that it's a full experience, especially if you only take a casual glance. but as much as it kills me (and believe me, it does), it isn't. not yet. not quite yet. Todd ACE
  5. Server Stability Update

    The sad thing is: he's not wrong. It really does seem like it should be easy! It's one of the most frustrating things about being a game developer, honestly... the enormous gap between "the amount of effort a reasonable person would think something should take" and "the amount of effort it actually takes". It's an order of magnitude (or more) effort to do every little thing. And as the project gets bigger and bigger, it gets worse! Back in the hunger dome days, with a much smaller art cache and code base, the simple act of adding a piece of art or a single line of code was like 20x faster than it is now... just as a simple function of compile time and file transfer time, not to mention complexity. "Making a build" that pulls all of the pieces together is something like 4+ hours on a wicked fast build server. It's crazy. but the only way to understand is to live through the process -- at which point you're on the other side of the line and everyone else is looking at you wondering why you can't get stuff done faster. ;p Disclaimer: YES, I absolutely realize that if that is one of the most frustrating things about my job, I must be a lucky s.o.b. with a pretty great job. Don't ever think I'm not thankful! Todd
  6. Server Stability Update

    I absolutely appreciate this. Your time is incredibly valuable as well, and I love the passion. That's part of the reason I am so bulll-headed about keeping our eyes on the prize of launch; I am desperate not to let you guys down on building this game and making it great. Todd
  7. Server Stability Update

    My text: "the smart, experienced people who have more information than you" You might want to re-read my post; there is a world of difference between what I wrote, and what you apparently read. No insult (or comparison) was intended; I was only asking that you reassess your approach: rather than assume we are making poor decisions (i.e. we didn't see or choose the easy and obvious solution), perhaps jump first to the assumption that we are privy to information you are not, and that if we came to a different conclusion, we probably had a good reason? Todd
  8. Server Stability Update

    I try, but to be clear: I've spent the last 1-2 hours this morning on the forums, trying to communicate, instead of working on the Campaign Management Tool design doc (which was my actual goal for the morning). That's absolutely fine, and I'm glad that I did (I consider it to be a part of my job, in fact), but I'm just saying: everything has a cost. Todd
  9. Server Stability Update

    The server cap code is deprecated, meaning that it hasn't been kept up to date (this is generally known as "code rot"). To get the caps working again, someone would need to go back and fix it -- not a huge task, I wouldn't think, but that code is part of the login / zoning flow, which has been completely rewritten in 5.5. so any effort spent fixing it in 5.4 would likely have to be redone in 5.5. If something seems easy and obvious, and yet the smart, experienced people who have more information than you haven't already done it, maybe you should question your base assumption (that it is easy and obvious)? We can prop more servers, though -- GW and Bill and I discussed that this morning in our leads meeting. We'll go ahead and do that; hopefully it will spread the player load and help a bit. Todd
  10. Server Stability Update

    Sorry, missed your last question. Some of these issues don't become apparent on TEST, because it doesn't always get enough load or doesn't have a long enough lifespan for latent issues to "gestate". and you're right that we are moving too fast; in a real live environment (post-launch) I sincerely hope that we won't be so quick to pull the trigger and say "it's good enough, move it to live." (and in our defense, we do get an inordinate amount of pressure to move stuff from test to live as quickly as possible, because test always has the "new hotness" and everyone wants to play it now-now-now). ;p Todd
  11. Server Stability Update

    "Why not just do a rollback?" Well, the short answer is: there is no short answer to that question. We're constantly discussing coping mechanisms to balance the two needs: keep the game as playable as possible, but try to spend 99% of our effort on forward momentum. There are some moments, like now, when it's particularly challenging because the longer a version is "live" the more the code base will have diverged and the harder it is to support that aging branch. "Do a rollback" is one such coping mechanism. That said, even coping mechanisms aren't free. "but just do a rollback!" makes it sound like that's an easy thing; you just hit the rollback button and the live environment morphs into a previous (presumably more functional) version. In actuality, we've moving so fast in development that a rollback is often incredibly challenging, if not impossible. In just the last 60 days, off the top of my head, we have: updated to a new unity version, upgraded to a new persistent database, changed out the web store (which changes the API for redeeming items), and reconfigured our AWS installation. We've also added I-couldn't-guess-how-many changes to art assets in the game (hundreds? thousands?), many of which would be incompatible with the old feature set ("can't load this zone, the old code don't recognize this new XYZ spawner marker") and radically changed both the structure of, and the data in, hundreds of design tables (including some tables that have been added, gutted or completely restructured.) Rolling back to a previous version would be a massive fire-drill that would affect... (guessing)... 50% of the company for... half a week? a week? Sometimes a rollback is straight-forward and sometimes it isn't. In this case, the "last version" of the game that didn't surface this issue had server crashes. so even if we did it, would that version be better? maybe? ...and, to my point earlier: while this fire drill might result in a more playable experience for testers (hooray!) it would come at a significant cost... and it wouldn't actually put us one inch closer to launch (boo!) A few weeks from now, we'd be dealing with the next round of bugs in the new version and today's victory would be forgotten. I absolutely understand your frustration. But I don't know what I can, or should, do different than we are right now. We're constantly looking at the issues and we're making calls every day on where to put our limited resources. If this version is too frustrating, I totally get it, and I can only say that I'm sorry and that we are working as hard as we can to get you a better version as soon as possible. Todd ACE
  12. Server Stability Update

    If I could wave a magic wand and fix it for free, I would. but obviously, I can't. Ultimately this is about priorities; where do we spend our limited resources. If we have time/money for XXX days of development, should we really spend any of those days fixing bugs in code that has already been replaced / no longer exists? That's the choice: the engineers can continue to support code that no longer exists -- making the game better right now, but ultimately taking time and money away from the product we're actually shipping -- or they can push forward to get the features down and then address stability and performance. Obviously it's not as simple as doing one or the other. We'll always have to do the best we can to server both masters.... but ultimately "get the game done" is have to be our top priority. Todd ACE
  13. Server Stability Update

    The team is primarily focused on trying to bring 5.5 to a close; we set a goal at the beginning of the year to increase the rate at which we put out major updates. That effort (and the recent hires) seems to be bearing fruit, but it does mean that our focus is on new development over fixing issues in the "live" build. One unfortunate side-effect is that some live issues will linger, because the live environment is a different code branch, and supporting that code branch often means fixing bugs in code that has already been removed/replaced in the new branch. Is it worth it to spend 2 days having a programmer fix a bug in code that no longer exists? In my opinion, no. Those 2 days would have been better spend marching towards feature complete and launch. We're choosing to put those resources on forward momentum. We are doing the best we can to serve both masters, but driving forward to launch is (and should be) our first priority. As we get closer to feature complete, you'll see our focus shift to stability and optimization (fps, latency, server load, etc). Todd ACE
  14. yeah it's been hard for me to keep up with the forums. as you know (but others might not) though I'm pretty active on twitter, so that's another avenue if you're trying to bring something to my attention. Todd
  15. I don't like to talk about stuff when it's still being figured out. That's one of the most unfortunate aspects of a crowdfunded projects; you have to make bold prognostications about systems and features long before you start working on them, and inevitably things change (technical limitations, resources available, design underpinnings, whatever) that force changes or open up new opportunities. Sometimes those changes are good (like finding the resources to move from archetypes to Race/Class) and sometimes they are painful (crap! the strongholddesign was based on the assumption that building placement would use lots, and we ended up more freeform placement. Now we need to add "tokens" to limit the number of buildings per parcel.) I've taken the philosophy that, ultimately, my job is to give you guys the best game that I can. If that means we have to change things, we change them. Some people will get upset because they don't agree with a particular change, but if the game is good that's what really matters. (And conversely, if we check every box exactly as stated at the beginning -- but the game is terrible -- we're screwed.) So by that logic, my #1 priority has to be the quality of the game experience. Todd ps. yes, this will be the topic of a future Tu/Th update.