Jump to content
Search In
  • More options...
Find results that contain...
Find results in...

Recommended Posts

Hey folks,

We put a new version up on Test last night -- this has a number of performance improvements, and a few known issues that we should probably highlight.

the Good

- First, we found a few improvements in the rendering pipeline.  This should give a frame rate boost on all machines.

- Grass cards have been improved to look better (no razor edges along the bottom) AND to perform better.  You can see the difference by turning your settings back to "Basic" ... but frankly, the overall performance improvement to this system means that you might as well run in Medium or higher.

- We added a new culling system to help performance as well.  "Culling" is a way of removing items and effects from the scene to improve framerate -- and, in theory, you shouldn't notice anything but increased performance if we are removing the right things.  (again, that's in theory).

- We fixed the framerate stutter that happened when you looked at harvesting interactables,

- Finally, we managed to get unity garbage collection under control! This one has been a long time coming, and what it means, in layman terms, is that you should see far less hitching as you run around the game.  Pay attention to the "SPK" (spike) counter in the upper left; you'll probably see it hit 4 or 5 when you first log in, as the world loads around you, and then it should settle down and stay at or near 0.  That's the hope, anyway, so let us know if that is what you are seeing.

the Bad

- In practice, the culling algorithm is making decisions for you about what you do/don't want to see, and algorithms are notoriously prone to error.  We've exposed settings for this in the preferences screen, so please try out different settings to cull things more / less aggressively and let us know if you can find the sweet spot.

- Eventually, we need the culled objects to fade in and out of view rather than just popping.  Once fading is in, the culling of objects will be less noticeable. 

(We have more advanced culling in the works, as well, but even this first iteration should pay dividends!)

-  Some items that were created in previous builds look like they have bad mouse-over information.  Mouse-over popup info for new items is working fine, and the old items are (I think?) just as functional as before, but the info being shown for them is incorrect.  The easy fix for this issue would be to wipe items, but that kind of sucks for testing, so let's see if we can hold off on doing that unless it becomes absolutely necessary.

& the Ugly

- There is a bug (potentially related to culling, but maybe not?) where you collide with something on the server that you don't see on the client, and after a moment it causes your character to pop/sync/teleport.  This is especially bad around outposts and hero statues/respawn locations, two locations where I'm sure you're really delighted to be unexpectedly yanked through space.

- Frostweaver targeting seems borked. 

- There is a particular power that seems to be causing a server exception.  Bad, bad, bad.

- Some state seems to be causing powers to stop working in a zone? This bug seems wiley, but thankfully rare.  

- There was an edge case that caused wall upgrades to be lost on a server reboot.  We believe this issue is now resolved, but please be on the look-out and let us know if it happens again.

In all of the above cases, we're looking at the Ugly bugs first, and we'll move on to the Bad once those are done.  As always, we working to get the issues resolved as quickly as possible.

"Why did you prop it, with known issues?"

That's a great question, thanks for asking.

Two answers to that: first, we don't always know all the issues -- or, we sometimes know an issue is there (as we did with teleport) but it turns out to be much worse in practice than we expected it to be.  and second, sometimes we don't know about the issues until the build is tested in front of real players.  Crowfall is a pretty beefy game at this point, with a lot of interlocking systems, and even with internal testing + our overseas testing firm that does a full set of tests every night while we are sleeping, we still don't catch every issue every time.  There isn't enough time in a 24 hour cycle to touch every single [ power, recipe, loot roll, monster, victory card, building, interactable, etc. ] in the game.

It's probably worth reminding everyone that this is the Test server, and there are always bugs on the Test server.  The more tolerant we are of pushing builds out to Test with known bugs, the more builds we can push!   and the opposite is true, also: if we want to ensure more quality, it means we prop less often.  In this case, we've gone several weeks since our last update, and that means hundreds of fixes waiting in the wings... and in the meantime, no way for players to harvest rare resources.  It seemed like a good idea to get it out and let you guys test it over the long weekend, even at the risk of the overall experience not being 100% playable.

If you find these kind of issues too frustrating -- and believe me, I can certainly understand if you do -- then I would caution you to wait until the game migrates to Live -- where the issues will tend to be bad rather than ugly.  For those of you who are willing to help us test, know that we appreciate it greatly, as the faster we can find the fixes the faster we can get them resolved and get this thing out to Live.

Hope that helps shed a little light on the method to our madness, and see you in Test, if you brave it... and if not, I hope you have a great memorial day weekend!

Todd

 


J Todd Coleman

ArtCraft Entertainment, Inc.

Follow us on Twitter @CrowfallGame | Like us on Facebook

[Rules of Conduct]

Share this post


Link to post
Share on other sites

@jtoddcoleman

Are there any particular issues or types of issues that ACE can use additional help with via in-game testing?  There are quite a few "volunteer" bug-investigators playing, typically operating with help from our guilds.  If there are particular areas to focus on that might increase the rate of some bug fixes, that would be of help to us, and also we would know to move on from bugs that you feel have been addressed in the unreleased dev builds.

Share this post


Link to post
Share on other sites
46 minutes ago, jtoddcoleman said:

- There is a bug (potentially related to culling, but maybe not?) where you collide with something on the server that you don't see on the client, and after a moment it causes your character to pop/sync/teleport.  This is especially bad around outposts and hero statues/respawn locations, two locations where I'm sure you're really delighted to be unexpectedly yanked through space.

this is very annoying, i'm glad it's a top priority :D

I hope you also will find why the update was partial yesterday :(

Share this post


Link to post
Share on other sites
4 minutes ago, Dern said:

@jtoddcoleman

Are there any particular issues or types of issues that ACE can use additional help with via in-game testing?  There are quite a few "volunteer" bug-investigators playing, typically operating with help from our guilds.  If there are particular areas to focus on that might increase the rate of some bug fixes, that would be of help to us, and also we would know to move on from bugs that you feel have been addressed in the unreleased dev builds.

...it's a great question, we tend to not give too much direction because (1) we need help with coverage on all the things, and (2) you guys aren't employees, so we hate to ask for things that feel like "task assignment".

that said, in this particular case, a lot of what we've been focused on are performance issues.  so any help with trying out the new culling system, and finding particular areas or situations that cause fps drops (other than the obvious, like "a ton of players using powers in the game area") would be greatly appreciated.

Todd

 


J Todd Coleman

ArtCraft Entertainment, Inc.

Follow us on Twitter @CrowfallGame | Like us on Facebook

[Rules of Conduct]

Share this post


Link to post
Share on other sites

Thank you, @jtoddcoleman !
 

We, the community, really appreciate you taking your time to inform us on what's going on and which challenges you are currently facing. We are passionate about the game and any piece of information is very valuable for us!

Most of players I communicate with are Okay with bugs because we do understand what the TEST server concept is and we gladly report those bugs back to you (the last bug report post on the forum had 6 or 7 pages in it, if I'm not mistaken).

The only thing I would ask you for is to announce the upcoming updates/campaigns a little bit in advance so that guild management could activate its members. Yesterday, when the new Dregs campaign was announced (only 1-2 hours before the siege time) we, testers/guild members, had  to urgently rush and change our plans on the go. A lot of people could not make it at all and have already made plans for the whole long weekend. And this is only 4-day campaign...

On the other hand, after having played the whole night in the new campaign, I did see a HUGE improvement in performance. The game now feels so much smoother and I would say that 70% of freezes and lags have gone away (didn't test it in mass fights and high graphics yet though). We appreciate your hard work in making this unique game and we will continue to help you with testing it. Just continue to keep us informed, please. We really value that 😃 

Share this post


Link to post
Share on other sites
50 minutes ago, SAM_BUKA said:

The only thing I would ask you for is to announce the upcoming updates/campaigns a little bit in advance so that guild management could activate its members. Yesterday, when the new Dregs campaign was announced (only 1-2 hours before the siege time) we, testers/guild members, had  to urgently rush and change our plans on the go. A lot of people could not make it at all and have already made plans for the whole long weekend. And this is only 4-day campaign...

Let's talk about that.

Internally, we cut new builds all day, every day.  There isn't just "one version" of the game internally; we have different groups working on different builds (like right now we have a "culling version" and "new player experience version" and "fix powers" version) and these are all duplicates (branches) of the game that different teams work on independently and we can bringing them up separately and testing them in separate environments. 

You have to have separate development tracks because otherwise when someone is doing open-heart surgery on one system, everyone else would have to wait.  obviously working linearly is not realistic for a team of 60+ people, we can't take turns doing tasks!  But this approach, with multiple branches, means every now and then we have to crunch those versions back together, which causes conflicts that we have to fix.  The longer you go without merging branches, that the more conflicts you find.  

So we are in a constant state of branching off new builds and then working independently, and then pulling them together and testing them and asking "does this pass the sniff test to prop to Test?" and that's a judgement call, every time.  

So, when we find a version that we believe (hope) passes the Test -- let's say, at 5:30pm on a Thursday leading into a weekend -- what do we do?  We could hold it until tomorrow (Friday) in which case if any bugs are found, which is very likely, there will be no one around to fix them (on Saturday).  Or we can prop it Thursday night, find the bugs that weekend, and have a chance -- a window -- to try and resolve those issues on Friday so that people can test over the weekend.

The only way to give you more notice is to hold the build.  and remember, this is Test (not Live) the purpose of propping the build is testing (not playing).

It seems to me that the right call, the priority, is to get the build out.  If your group is treating this as a campaign that they want to win, I suggest you might want to temper those expectations...?  Test server is going to be buggy.  It's going to go up and down as needed, to test.  We might end campaigns early.  We may wipe at any time.  Treating it like the Live server is, IMO, a dicey proposition.

Todd

 


J Todd Coleman

ArtCraft Entertainment, Inc.

Follow us on Twitter @CrowfallGame | Like us on Facebook

[Rules of Conduct]

Share this post


Link to post
Share on other sites
54 minutes ago, Aedius said:

I hope you also will find why the update was partial yesterday :(

Aha, this is helpful because I can leverage my previous post to answer!  See my note above about branches -- the patch notes were erroneously generated to include some fixes that were in branches not included in this particular merge.

Todd

 


J Todd Coleman

ArtCraft Entertainment, Inc.

Follow us on Twitter @CrowfallGame | Like us on Facebook

[Rules of Conduct]

Share this post


Link to post
Share on other sites
9 minutes ago, jtoddcoleman said:

So we are in a constant state of branching off new builds and then working independently, and then pulling them together and testing them and asking "does this pass the sniff test to prop to Test?" and that's a judgement call, every time.  

Can you tell us when you are pulling them togheter and if you want to bring a campaign up with this pull.A couple of us didn't expect a dregs campaign yestarday cause from our pov half of the patch was missing so we were surprised to see a dregs campaign up.Even if the pull doesn't pass the sniffing test thats better than having one out of the blue as long as you are comunicating with us.We don't have a problem if you tell us "Hey we are trying to push an update today, we will tell you in x time if it will be pushed or not,if its pushed we also want to start a campaign." That way we know when to expect the update and if we should expect a dregs.

Share this post


Link to post
Share on other sites

😂 Thank you for shedding some light on all this Todd. We as testers understand what is going on better now, letting us know like you have just done helps everyone's moral quite a bit. I know a lot of people would love a heads up, but my wife and I have no complaints lol. It is very much appreciated! Some of us are "playing to win" while more so others are (hopefully) testing every possible mechanic and not just PvP related and performance issues. There are even some of us that do the more tedious tasks of hitting rocks/trees/ore/skinning then crafting it, to make sure everything is a smooth transition and is operating normally. The only downside to this which could of been said a little bit better, is that roadblocks like no campaigns for a couple weeks really hurts the progress is all 😃. By progress I mean non-PvP related progress lol. Regardless my wife and I continue to test all aspects of harvesting and crafting (with a little PvP fun thrown in) to the best of our ability, and we just want to say thank you. We are just happy to be testing and playing, if there is anything else we can do to make this push to live let us know (even if it is "task assignment"). We would be more then happy to do so!

Share this post


Link to post
Share on other sites
Posted (edited)
52 minutes ago, jtoddcoleman said:

It seems to me that the right call, the priority, is to get the build out.  If your group is treating this as a campaign that they want to win, I suggest you might want to temper those expectations...?  Test server is going to be buggy.  It's going to go up and down as needed, to test.  We might end campaigns early.  We may wipe at any time.  Treating it like the Live server is, IMO, a dicey proposition.

Todd

 

@jtoddcoleman thank you. I really appreciate you taking your time answering our questions today. Your answers have really shed some light on what is going on behind the scene and that will eliminate many questions and sad feelings inside the community, for sure.

At the same time, as you mentioned earlier in your answers: we (testers/players) are not employees and we are not robots either. We still want to play and enjoy the game even at it's current state. As much as we want to help ACE with the feedback and bug reports (which we always do), and as much as we understand that bugs are inevitable on TEST server and sudden wipes are possible... we still like to PLAY the game and enjoy it. And yes, we want to win. Crowfall is about winning, isn't it? That's why we play and test it. And these two terms "Play & Test" will always come together if you want to involve your backers in the game development process and receive live feedback.

It seems like this fact is important to understand from the developers end, as well. As much as it is important for us, players, to understand that the game is still in development.

Thank you for your time, Todd, once again! 🙂

Edited by SAM_BUKA

Share this post


Link to post
Share on other sites
Posted (edited)
3 hours ago, jtoddcoleman said:

t's probably worth reminding everyone that this is the Test server, and there are always bugs on the Test server.  The more tolerant we are of pushing builds out to Test with known bugs, the more builds we can push!   and the opposite is true, also: if we want to ensure more quality, it means we prop less often.  In this case, we've gone several weeks since our last update, and that means hundreds of fixes waiting in the wings... and in the meantime, no way for players to harvest rare resources.  It seemed like a good idea to get it out and let you guys test it over the long weekend, even at the risk of the overall experience not being 100% playable.

 

It seems like the team is at odds with providing an in-game experience vs bug testing on the test server. My suggestion, treat it like a test server and inform the players that it is for testing and provide potential areas of testing for the community. Stand-up a simple MOTD system that has notes and areas of testing, and use this time to have those engaging in the test server do so. Do not make this seem like a live environment or it will result in players getting upset, as is already happening. Make it easy to acquire item again if the purpose isn't to test "farming" but to fix bugs and test the combat systems. The more people are spending hours farming and engaging with content that may not be the focus of the testing is going disrespect the players time.

To me there is no clear testing priorities that should be tackled by the active players and the test server is not being conveyed as test server. Keep stating over and over "THIS IS TEST! IT WILL HAVE BUGS! HELP US TRACK XYZ! REPORT LIKE THIS! WE WILL FIX IT!"

Edit:

I probably seemed to target ACE here but honestly I would say the same about many of the players currently on test. This is a test server for a game so you need to treat it as such. It is not the live game and it is not meant to be treated as such. Making games is difficult and ACE is trying to listen to feedback but I see a lot of complaints at ACE that are not informative or beneficial (not all, there are some good ones). You need to state your mind but always do it in a constructive kind of way. Provide solutions and don't be condescending. Game developers are much more receptive to feedback when you make suggestions not demands.

Edited by KanashiGD

Game Designer | KanashiGD.com | @KanashiGD

Elf of Hy'shen Avari

Share this post


Link to post
Share on other sites

A fairly consistent problem in the rendering pipeline still exists when loading anims (or models + anims) that cause long frame times (hitching).
Anecdotally, players notice this while roaming due to loading npc anims that are popping in view range, and note that the problem is less severe during night time when view range is reduced.
This is also at least one of the problems related to freezing up immediately on engaging with an enemy group, as they are just coming into view and loading these resources.
The anims are loading all the way from disk, according to Windows resource monitor, and do not appear to be preloaded in RAM. The problem is also worse on lower memory systems if for some reason these (and other) assets have been paged to disk despite being loaded prior.

@yianni may have relayed some of this information previously when I was explaining my black box take on it to others in our Discord.

For reference, I'm working with: i7-4790k, 16GB, Nvidia 980GTX. Capping at 40fps.
Driver settings for extremely reduced texturing
Antialiasing - Transparency Supersampling "AA_MODE_REPLAY_MODE_ALL"
Texture filtering - LOD Bias (DX) "0x00000078"

The following data is captured with FrameView while roaming outside the temple. MsBetweenPresents is the notable frame time stat, these are the worst entries, associated with anim load.

Application	SwapChainAddress	Runtime	SyncInterval	PresentFlags	AllowsTearing	PresentMode	Dropped	TimeInSeconds	MsBetweenPresents	MsBetweenDisplayChangeActual	MsUntilRenderComplete	MsUntilDisplayed
CrowfallClient.exe	0x00000203C850D4B0	DXGI	1	0	0	Composed: Flip	0	194.00572	7593.057	7652.779	117.02	135.335
CrowfallClient.exe	0x00000203C850D4B0	DXGI	1	0	0	Hardware: Independent Flip	0	114.593039	2368.404	2354.169	5.777	6.628
CrowfallClient.exe	0x00000203C850D4B0	DXGI	1	0	0	Hardware: Independent Flip	0	112.224635	1705.997	1659.716	15.734	20.863
CrowfallClient.exe	0x00000203C850D4B0	DXGI	1	0	0	Hardware: Independent Flip	0	116.616614	1106.306	1069.434	2.681	3.868
CrowfallClient.exe	0x00000203C850D4B0	DXGI	1	0	0	Hardware: Independent Flip	0	128.803181	879.135	826.381	23.794	25.593
CrowfallClient.exe	0x00000203C850D4B0	DXGI	1	0	0	Hardware: Independent Flip	0	127.570043	622.965	618.027	1.568	1.794
CrowfallClient.exe	0x00000203C850D4B0	DXGI	1	0	0	Hardware: Independent Flip	0	115.510308	381.669	347.225	36.48	40.74
CrowfallClient.exe	0x00000203C850D4B0	DXGI	1	0	0	Hardware: Independent Flip	0	196.601643	221.07	229.16	21.012	25.493
CrowfallClient.exe	0x00000203C850D4B0	DXGI	1	0	0	Hardware: Independent Flip	0	129.211116	217.58	222.23	21.223	27.398
CrowfallClient.exe	0x00000203C850D4B0	DXGI	1	0	0	Hardware: Independent Flip	0	196.380573	195.578	152.793	14.895	17.403
CrowfallClient.exe	0x00000203C850D4B0	DXGI	1	0	0	Hardware: Independent Flip	0	128.993536	190.356	187.51	20.915	22.748
CrowfallClient.exe	0x00000203C850D4B0	DXGI	1	0	0	Hardware: Independent Flip	0	126.947078	174.273	104.189	1.374	6.732
CrowfallClient.exe	0x00000203C850D4B0	DXGI	1	0	0	Hardware: Independent Flip	0	129.37307	161.954	159.716	21.087	25.16
CrowfallClient.exe	0x00000203C850D4B0	DXGI	1	0	0	Composed: Flip	0	194.147262	141.541	97.201	67.352	90.995
CrowfallClient.exe	0x00000203C850D4B0	DXGI	1	0	0	Hardware: Independent Flip	0	194.595592	123.204	83.338	22.63	24.607
CrowfallClient.exe	0x00000203C850D4B0	DXGI	1	0	0	Composed: Flip	0	37.771831	108.582	104.164	7.054	15.657
CrowfallClient.exe	0x00000203C850D4B0	DXGI	1	0	0	Hardware: Independent Flip	0	196.804258	104.859	104.169	7.327	10.366
CrowfallClient.exe	0x00000203C850D4B0	DXGI	1	0	0	Hardware: Independent Flip	0	200.989641	100.01	48.624	7.549	12.47


A normal/optimal frame looks like (25ms frame time * 40 frames -> 40 frames per 1 second)

CrowfallClient.exe	0x00000203C850D4B0	DXGI	1	0	0	Hardware: Independent Flip	0	119.846177	25.06	27.782	73.079	79.856


Further notes for other players

By default, the game engine trys to render as many frames as possible, or to as many frames as vsync if that option is checked.
Going from very small frame times (high fps) to a suddenly long frame time due to loading assets causes a longer frame time than if you cap your fps.
Capping fps also prevents cpu from being fully maxed and may also allow the thread that is loading the anim asset from disk a bit of breathing room... maybe.
If your CPU cooling solution sucks (or dust buildup), you could also suffer from CPU throttling due to temperature.

Frame times explained
https://www.youtube.com/watch?v=nbHCU--VvpQ

Share this post


Link to post
Share on other sites
2 hours ago, jtoddcoleman said:

Let's talk about that.

Internally, we cut new builds all day, every day.  There isn't just "one version" of the game internally; we have different groups working on different builds (like right now we have a "culling version" and "new player experience version" and "fix powers" version) and these are all duplicates (branches) of the game that different teams work on independently and we can bringing them up separately and testing them in separate environments. 

You have to have separate development tracks because otherwise when someone is doing open-heart surgery on one system, everyone else would have to wait.  obviously working linearly is not realistic for a team of 60+ people, we can't take turns doing tasks!  But this approach, with multiple branches, means every now and then we have to crunch those versions back together, which causes conflicts that we have to fix.  The longer you go without merging branches, that the more conflicts you find.  

So we are in a constant state of branching off new builds and then working independently, and then pulling them together and testing them and asking "does this pass the sniff test to prop to Test?" and that's a judgement call, every time.  

So, when we find a version that we believe (hope) passes the Test -- let's say, at 5:30pm on a Thursday leading into a weekend -- what do we do?  We could hold it until tomorrow (Friday) in which case if any bugs are found, which is very likely, there will be no one around to fix them (on Saturday).  Or we can prop it Thursday night, find the bugs that weekend, and have a chance -- a window -- to try and resolve those issues on Friday so that people can test over the weekend.

The only way to give you more notice is to hold the build.  and remember, this is Test (not Live) the purpose of propping the build is testing (not playing).

It seems to me that the right call, the priority, is to get the build out.  If your group is treating this as a campaign that they want to win, I suggest you might want to temper those expectations...?  Test server is going to be buggy.  It's going to go up and down as needed, to test.  We might end campaigns early.  We may wipe at any time.  Treating it like the Live server is, IMO, a dicey proposition.

Todd

 

@jtoddcoleman Multiple teams working on individual feature branches and merging into a release branch has always been a thing and is part of standard development practice in every industry.  What concerns me is that externally it appears that a high percentage of tests you conduct are manual and there is an insufficient test harness in place to automatically check for regressions.  As we move more toward beta and you guys decide what features you want in beta candidate for the game, will a higher percentage of sprint time be put towards expanding and building out a test framework for catching lower level regressions (disciplines not working as intended, the repeated issues with druid orbs, etc)?

Share this post


Link to post
Share on other sites

@jtoddcoleman the engagement and info you've shared in this thread here is all a lot of people are really asking for. It is really great. I know it takes time of your away from really important aspects of the company and development team management, but I'm sure tons of the old guard faithful love seeing this. So thank you. Have a nice holiday ACE

Share this post


Link to post
Share on other sites

From @jtoddcoleman at Everyone....

"CHECK PLEASE" 🤣

honestly this is TEST and you are right Todd I think we seem to forget its not meant for the "play to win mentality". Its play to test more so test then play. We knew what we signed up for and I'm grateful to still be able to help you all test. "God"(jtodd)has spoken. This is the way....

 

Share this post


Link to post
Share on other sites
22 hours ago, SAM_BUKA said:

The only thing I would ask you for is to announce the upcoming updates/campaigns a little bit in advance so that guild management could activate its members. Yesterday, when the new Dregs campaign was announced (only 1-2 hours before the siege time) we, testers/guild members, had  to urgently rush and change our plans on the go. A lot of people could not make it at all and have already made plans for the whole long weekend. And this is only 4-day campaign...

This is TEST in an ALPHA game. They don't have to announce anything. They don't have to cater to any guilds "plans" or "schedule". 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...